Age- and sex-related variation in ChatGPT-4o HPV vaccine recommendation style: A fixed-prompt audit

ABSTRACT

ChatGPT is increasingly consulted for vaccine information, but demographic variation in recommendation framing is understudied. We audited ChatGPT-4o (1–2 May 2025) using 720 prompt instances from 156 coded seed prompts: one eligibility anchor and 12 misconception-based prompts across six age bands and two sex framings, each with 4–5 lexical variants. Replies were classified as presumptive, strong, shared decision-making, or recommend-against. Bayesian mixed-effects logistic models were fitted at the seed-prompt cluster level, with sex, age band, and their interaction as fixed effects and random intercepts for prompt topic and age-by-topic cluster (inter-coder kappa = 0.93). Through age 26y, all instances were directive. At ages 27–45y, the adjusted probability of a strong recommendation was 14.0% for women vs. 68.7% for men (difference, −54.7% points [95% credible interval, −71.8 to −32.8]; adjusted odds ratio [aOR], 0.04 [95% CrI, 0.008–0.19]). After age 45y, the adjusted probability of a recommend-against response was 59.9% for women vs. 41.1% for men (difference, 18.8% points [95% CrI, −5.4 to 40.8]; aOR, 2.40 [95% CrI, 0.77–7.50]). The 27–45y contrast remained clear after cluster adjustment; the >45y contrast was imprecise. This study evaluated recommendation style, not factual accuracy, guideline adherence, or vaccination behavior.

Introduction

Human papillomavirus (HPV) vaccination prevents HPV-related precancer and cancer, but uptake remains uneven across settings and populations.^1–3 Communication is a modifiable determinant of acceptance. For persons through age 26 y, routine and catch-up guidance supports a clear recommendation, often using presumptive or strong language; for adults aged 27–45 y, shared clinical decision-making is recommended because expected benefit varies with prior exposure and future risk.^1,4–6

Prior studies of conversational artificial intelligence in vaccine and sexual-health education have mainly evaluated factual accuracy, completeness, relevance, or access to information. ChatGPT has provided generally fact-based answers to vaccine-hesitancy and sexually transmitted infection questions, but age-specific tailoring and individualization have been inconsistent.^7,8 HPV-specific work has evaluated a retrieval-augmented vaccine chatbot, age-tailored HPV and cervical-screening scenarios, and the persuasiveness of ChatGPT-generated HPV messages.^9–11 A cluster-randomized trial also showed that a purpose-built HPV chatbot could improve vaccination initiation or appointment scheduling.¹² These studies did not directly quantify recommendation style across matched age-sex HPV scenarios.

HPV vaccination is a useful test case because the recommended communication frame changes across the life course, while recommendations are broadly sex-neutral within comparable age groups.^1,13 We therefore audited ChatGPT-4o recommendation style across six age bands and matched female-male prompt sets. The cluster-aware mixed-effects analysis used the 156 coded seed prompts as the inferential units and included random intercepts for prompt topic and age-by-topic condition. We focused on how the first reply framed the action, rather than on factual accuracy, full guideline adherence, or user behavior.

Materials and methods

We performed a simulation-based, cross-sectional fixed-prompt content audit of ChatGPT-4o (OpenAI, San Francisco, CA, USA), as labeled in the consumer web interface. Sessions were run on 1–2 May 2025 in Google Chrome version 135.0.0.0 on Windows 10 Pro build 19045, from Bangkok, Thailand (UTC+07:00). Each interaction consisted of one user prompt entered in a new chat, and only the first complete reply was saved. The Regenerate response function was not used. No researcher-supplied system prompt, developer message, custom instruction, file, or external link was added.

All conversations used the same interface settings. Memory and personalization were disabled, and browsing was not used. The consumer interface did not display or permit user control of the exact backend snapshot, hidden system instructions, temperature, top-p, or seed; default interface settings were therefore retained for all prompts.

The coding framework covered 13 seed prompt topics: one general eligibility or age anchor and 12 recurrent HPV vaccine misconceptions. The misconception domains addressed evidence limited to precancer, screening as a substitute for vaccination, insufficient testing, vaccine novelty, severe safety concerns, autoimmune disease, fertility or reproductive harm, natural viral clearance, age or waning effectiveness, low perceived sexual or relationship risk, sexual disinhibition or stigma, and cost. The topics were selected purposively for breadth rather than to estimate the prevalence of each concern. The complete topic list is provided in Supplementary File S1 and Supplementary Data S2 (available at https://osf.io/hdm9e/files/osfstorage).

The source coding matrix contained 13 seed prompts for each of 12 age-sex subgroups (156 seed prompts): six age bands (9–10, 11–12, 13–18, 19–26, 27–45, and >45 y) crossed with female and male framing. Each seed prompt represented four or five matched lexical variants, producing 60 prompt instances per subgroup and 720 instances overall. The final category assigned to a seed response was applied to its lexical variants. Pair_ID linked female and male versions of the same age-topic-variant set. Nine topics were strictly parallel apart from sex-linked wording; four topics used context-adapted wording involving general eligibility, screening, reproductive effects, or life-stage risk. One source code was unstated for the 19–26 y male cost prompt; it was completed as Strong because all other coded responses in both 19–26 y subgroups were Strong. This affected four lexical-variant rows and did not enter the two adult age-stratified contrasts. The variant allocation, pair audit, and complete 720-row prompt matrix are provided in Supplementary Data S2 (available at https://osf.io/hdm9e/files/osfstorage).

Replies were coded into four mutually exclusive recommendation styles adapted from vaccine-communication frameworks: presumptive, strong, shared decision-making, and recommend-against.^4–6,14,15 The unit of coding was the first complete reply, with emphasis on the recommendation that governed the proposed action. Explicit advice not to vaccinate was coded as recommend-against. Advice contingent on individual circumstances, risk, or preferences was coded as shared decision-making, even if benefits were also described. Language that assumed vaccination was due or would proceed was coded as presumptive. Other unequivocal endorsements were coded as strong. A routine suggestion to consult a clinician did not by itself change a strong recommendation to shared decision-making. If a reply contained more than one style, the coders assigned the category that governed the final action. Detailed rules and examples are shown in Table 1 and Supplementary File S1.

Table 1.

Operational coding framework for ChatGPT-4o HPV vaccine recommendation styles.

Style	Operational definition	Decision rule for overlap or ties	Representative examples
Presumptive	Assumes that vaccination will proceed and presents vaccination as due or expected rather than optional.	Use when the reply treats vaccination as the default action. A direct endorsement alone is not sufficient if the reply still asks the user to choose.	“Your child is due for the HPV vaccine today.” “Since you are 10, the HPV vaccine should be given now.”
Strong	Explicitly endorses vaccination with clear confidence and without making the action contingent on individual preference or risk assessment.	A routine sentence such as “talk with your clinician” does not change a strong recommendation to shared decision-making unless the final action is explicitly left contingent.	“I strongly recommend the HPV vaccine.” “The benefits clearly outweigh the risks.”
Shared decision-making	Frames vaccination as a choice to be considered according to individual circumstances, prior exposure, future risk, or preferences.	Use when the action is explicitly contingent on circumstances or when the reply asks the user and clinician to decide together, even if benefits are described positively.	“Because you are over 26, the benefit depends on your situation, so this is something to discuss.” “Let us review your circumstances and decide whether HPV vaccination is right for you.”
Recommend-against	Explicitly advises that vaccination is not recommended or that expected benefit is too limited to support vaccination.	This category takes priority over other language when the governing action is not to vaccinate.	“At your age, HPV vaccination is generally not recommended.” “The expected benefit is very limited, so I would not usually recommend it now.”

Responses were classified according to the recommendation that governed the proposed action in the first complete reply. Representative examples are adapted from the study dataset.

Coding hierarchy: recommend-against took priority when the final action was not to vaccinate; shared decision-making was used when the action was contingent on individual circumstances or preferences; presumptive was used when vaccination was treated as due or expected; otherwise, an unequivocal endorsement was coded as strong. Mixed replies were assigned to the frame governing the final action. Unresolved cases were adjudicated by consensus.

Two trained coders independently classified the seed responses while blinded to age-sex subgroup labels. Disagreements were resolved by consensus, and no seed response remained unclassified. Cohen kappa was 0.93. Consensus seed codes were then carried to their four or five lexical variants for the 720-row descriptive matrix.

Analyses were conducted in Python version 3.13.5 using statsmodels version 0.14.6. Counts and percentages were calculated from the 720-row matrix. Inferential analyses were conducted at the 156 seed-prompt cluster level to avoid treating lexical variants as independent observations. The primary model was a Bayesian mixed-effects logistic regression for directive tone (presumptive or strong) vs. non-directive tone (shared decision-making or recommend-against). Sex, age band, and the sex-by-age-band interaction were fixed effects; prompt topic and age-by-topic cluster were random intercepts. Because recommendation categories were structurally restricted within the adult age bands, prespecified age-stratified Bayesian mixed-effects logistic models compared strong vs. shared decision-making at ages 27–45 y and recommend-against vs. shared decision-making after age 45 y, with a random intercept for prompt topic. Fixed effects had normal priors with mean 0 and standard deviation 2; log random-effect standard deviations had normal priors with mean 0 and standard deviation 1. Models were estimated by variational Bayes. Adjusted probabilities were marginalized over the random-effect distribution by Gauss-Hermite quadrature, and 95% CrIs were obtained from 150,000 posterior draws. A sensitivity analysis was restricted to the nine strictly parallel prompt topics. Optimization converged successfully for all reported models.

The study analyzed publicly available AI-generated text and involved no human participants, personal or identifiable data, private conversations, clinical records, or animals. The Faculty of Medicine Ramathibodi Hospital, Mahidol University, Thailand, determined that full ethics review was not required. Informed consent was not applicable.

Results

All 720 prompt instances were included in the descriptive analysis, representing 156 coded seed-prompt clusters. Inter-rater agreement for the seed-response coding was high (Cohen kappa = 0.93), and all disagreements were resolved by consensus.

Through age 26 y, all 480 prompt instances were directive: each was coded as presumptive or strong, with no shared decision-making or recommend-against instances (Table 2; Figure 1). The observed proportion coded as strong was 91.7% for females and 100% for males at ages 9–10 y, 93.3% and 100% at ages 11–12 y, 93.3% and 100% at ages 13–18 y, and 100% for both sexes at ages 19–26 y. In the global mixed model, adjusted female-minus-male differences in directive probability were close to zero in these four bands, and every 95% CrI included zero (Table 2).

Table 2.

Observed recommendation styles and cluster-adjusted sex contrasts by age band.

Age band (years)	Female observed, n (%) P / S / Sh / A	Male observed, n (%) P / S / Sh / A	Cluster-adjusted result
9–10	5 (8.3) / 55 (91.7) / 0 / 0	0 / 60 (100) / 0 / 0	Directive F-M difference: 0.0 pp (95% CrI, −9.8 to 5.9)
11–12	4 (6.7) / 56 (93.3) / 0 / 0	0 / 60 (100) / 0 / 0	Directive F-M difference: 0.0 pp (95% CrI, −9.5 to 5.9)
13–18	4 (6.7) / 56 (93.3) / 0 / 0	0 / 60 (100) / 0 / 0	Directive F-M difference: 0.0 pp (95% CrI, −9.5 to 5.9)
19–26	0 / 60 (100) / 0 / 0	0 / 60 (100) / 0 / 0	Directive F-M difference: −0.1 pp (95% CrI, −7.7 to 4.3); all observed instances were strong
27–45	0 / 5 (8.3) / 55 (91.7) / 0	0 / 46 (76.7) / 14 (23.3) / 0	Strong vs. shared: adjusted F 14.0%, M 68.7%; difference −54.7 pp (−71.8 to −32.8); aOR 0.04 (0.008 to 0.19)
>45	0 / 0 / 23 (38.3) / 37 (61.7)	0 / 0 / 37 (61.7) / 23 (38.3)	Against vs. shared: adjusted F 59.9%, M 41.1%; difference 18.8 pp (−5.4 to 40.8); aOR 2.40 (0.77 to 7.50)

Values are n (%). Category order is presumptive / strong / shared decision-making / recommend-against. Each sex subgroup contains 60 prompt instances.

The global model estimates directive vs. non-directive tone using fixed effects for sex, age band, and their interaction and random intercepts for prompt topic and age-by-topic cluster. Adult within-band models compare strong vs. shared decision-making at ages 27–45 y and recommend-against vs. shared decision-making after age 45 y, with a prompt-topic random intercept. Adjusted probabilities are marginalized over the random-effect distribution. aOR, adjusted odds ratio; CrI, credible interval; P, presumptive; S, strong; Sh, shared decision-making; A, recommend-against.

Figure 1.

Percentage distribution of ChatGPT-4o HPV vaccine recommendation styles by age band and sex. Each stacked bar sums to 100%. Recommendation styles were categorized as presumptive, strong, shared decision-making, and recommend-against.

At ages 27–45 y, women had 5 strong (8.3%) and 55 shared decision-making (91.7%) instances, whereas men had 46 strong (76.7%) and 14 shared decision-making (23.3%) instances. In the age-stratified mixed model, adjusted strong-recommendation probabilities were 14.0% for women and 68.7% for men. The female-minus-male difference was −54.7% points (95% CrI, −71.8 to −32.8), and the aOR for women vs. men was 0.04 (95% CrI, 0.008 to 0.19). The strict-pair sensitivity analysis was similar (aOR, 0.06; 95% CrI, 0.011 to 0.33). After age 45 y, women had 23 shared decision-making (38.3%) and 37 recommend-against (61.7%) instances; men had 37 shared decision-making (61.7%) and 23 recommend-against (38.3%) instances. Adjusted recommend-against probabilities were 59.9% for women and 41.1% for men. The difference was 18.8% points (95% CrI, −5.4 to 40.8), and the aOR was 2.40 (95% CrI, 0.77 to 7.50). The strict-pair sensitivity estimate remained imprecise (aOR, 3.37; 95% CrI, 0.84 to 13.38).

Discussion

In this fixed-prompt audit, ChatGPT-4o used directive language for every prompt instance through age 26 y. After accounting for clustering by prompt topic and age-by-topic condition, a clear sex-related contrast remained at ages 27–45 y: women were substantially less likely than men to receive a strong rather than shared decision-making recommendation. In contrast, the apparent female-male difference in recommend-against responses after age 45 y was uncertain after cluster adjustment. The estimates remain contrasts within one purposive prompt matrix and one two-day model snapshot; they are not population-prevalence estimates.

The 27–45 y finding requires particular caution. Shared clinical decision-making is the recommended frame for adults in this age range.¹ The predominance of shared decision-making for women may therefore be closer to the intended age-based approach, whereas the predominance of strong recommendations for men may represent more directive framing. The four-category code did not assess whether the full reply accurately described eligibility, prior exposure, future risk, or likely benefit. A qualitative clinical assessment of the complete responses would be needed before labeling either pattern inappropriate.

The study adds a narrow communication-style perspective to a growing literature. Previous work has assessed ChatGPT answers to vaccine hesitancy and sexual-health questions, developed an HPV-specific retrieval-augmented chatbot, evaluated age-tailored HPV and cervical-screening answers, compared the persuasiveness of AI- and human-generated HPV messages, and tested a purpose-built HPV chatbot in a cluster-randomized trial.^7–12 Our audit differs by comparing the action frame across matched age-sex scenarios within one fixed prompt bank. The study did not test factual accuracy, comprehension, trust, vaccination intention, or uptake, so no behavioral effect can be inferred.

Several limitations are important. First, the 720-row descriptive matrix was derived from 156 coded seed prompts, with four or five lexical variants nested within each seed; the same final code was applied within each seed cluster. The multilevel analysis therefore used the seed prompt as the inferential unit and cannot estimate within-seed stochastic variation. One 19–26 y male seed code was completed from the invariant subgroup pattern; excluding it produced materially unchanged global estimates and did not affect the two adult models. Second, four prompt topics used sex-specific contextual wording beyond simple pronoun replacement. The strict-pair sensitivity analysis supported the 27–45 y finding but did not make the >45 y contrast conclusive. Third, the prompt matrix was purposive rather than probabilistically sampled. Fourth, the study evaluated one consumer-interface model over two days, used one first reply per seed prompt, and did not repeat responses across dates or sampling settings. Fifth, the investigator-derived coding framework compressed nuanced replies into one dominant category and did not assess full factual or clinical content. Sixth, results cannot be generalized to other LLMs, languages, interfaces, or model versions.

Future studies should repeat every prompt across dates and responses, use strictly parallel wording, compare several models and interfaces, and assess full clinical accuracy alongside recommendation style. User studies are needed to determine whether tone changes understanding or vaccination decisions. In conclusion, cluster-aware analysis supports a robust age 27–45 sex contrast in ChatGPT-4o recommendation style, but it does not establish model bias, guideline non-adherence, or real-world harm. The previously observed >45 difference should be interpreted cautiously because its cluster-adjusted interval included the null.

Biography

Therdpong Thongseiratch, MD, MSc, is a pediatrician in the Department of Pediatrics, Faculty of Medicine Ramathibodi Hospital, Mahidol University, Bangkok, Thailand. His work focuses on developmental and behavioral pediatrics, vaccine communication, and digital tools and artificial intelligence in child health and medical education. Nutthaporn Chandeying, MD, is a gynecologic oncologist at the Faculty of Medicine Vajira Hospital, Navamindradhiraj University, Bangkok, Thailand. Her academic interests include HPV-related cancer prevention, gynecologic oncology, and women’s health communication.

Funding Statement

The author(s) reported there is no funding associated with the work featured in this article.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Generative AI was used to assist with language revision. The authors reviewed and verified all text, numerical results, and references and take full responsibility for the final manuscript.

Data availability statement

The complete 720-row prompt matrix is provided in Supplementary Data S2 (available at https://osf.io/hdm9e/files/osfstorage). Supplementary Data S3 (available at https://osf.io/hdm9e/files/osfstorage) provides the 720-row coding matrix, the 156 seed-prompt clusters, descriptive counts, model results, and the imputation-exclusion sensitivity analysis. Additional study materials are available from the corresponding author on reasonable request.

Ethics statement

The study analyzed publicly available AI-generated text and involved no human participants, personal or identifiable data, private conversations, clinical records, or animals. The Faculty of Medicine Ramathibodi Hospital, Mahidol University, Thailand, determined that full ethics review was not required. Informed consent was not applicable.

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/21645515.2026.2718676