Abstract
This quality improvement study investigates if a large language model could simulate adolescents’ responses to vaping-prevention campaigns and identify the most effective messages to address the public health crisis of adolescent vaping.
Artificial intelligence (AI) may not only improve patient care and clinical outcomes1 but also transform preventive medicine. Researchers currently spend millions of dollars—and participants spend thousands of hours—testing the persuasiveness of health messages. We performed 2 studies testing whether the large language module (LLM) GPT-4.0 (OpenAI) could simulate adolescents’ responses to vaping-prevention campaigns and quickly and efficiently identify the most effective messages to address the public health crisis of adolescent vaping.2
Methods
Both studies received institutional review board approval and obtained informed consent, followed the Standards for Quality Improvement Reporting Excellence (SQUIRE) reporting guidelines, and tested hypotheses via Pearson correlations conducted in SPSS, version 28.0 (IBM Corp).
Study 1
A database of 46 text-only messages (eg, “E-cigarette vapor can contain toxic metals”) were rated by adolescents who were susceptible to vaping in August 2022.3 These adolescent ratings were obtained using the 3-item Perceived Message Effectiveness (PME) scale, a reliable instrument that predicts motivational and behavioral responses to vaping communications.4 AI training involved describing the 3-item PME scale and seeding the LLM with adolescent PME ratings for 8 messages. The LLM then generated PME scores for the remaining 38 messages (eAppendix 1 and 3 in Supplement 1).
Study 2
LLMs are trained on a huge corpus of written material and may be especially suited for rating short, declarative statements. A stronger test of its human-simulation abilities involved a second, larger database comprising both words and images (eAppendix 2 and 4 in Supplement 1).
We collected 220 vaping-prevention ads used throughout the US from the Vaping Prevention Resource database at the University of North Carolina that were then rated by 649 vaping-susceptible adolescents in October to December 2020.5 The LLM was trained on 18 of these ads (8.1%) in 2 phases. First, we submitted the text from each ad and their PME ratings; second, we added descriptions of the ads’ imagery (eAppendix 5 and 6 in Supplement 1). These descriptions followed a structured protocol and described the type of ad (eg, poster), image characteristics (eg, illustration), and other visually striking features (eg, colors). The LLM generated PME ratings for the 202 ads in both phases.
Results
Participant characteristics are available in the Table.
Table. Characteristics of the Adolescent Samples in Studies 1 and 2.
| Characteristic | Study 1 (n = 656) | Study 2 (n = 649) |
|---|---|---|
| Age, mean (SD), y | 15.1 (1.36) | 15.3 (1.37) |
| Age category, No. (%), y | ||
| 13 | 96 (14.6) | 102 (15.7) |
| 14 | 152 (23.2) | 96 (14.8) |
| 15 | 120 (18.3) | 127 (19.6) |
| 16 | 153 (23.3) | 178 (27.4) |
| 17 | 135 (20.6) | 146 (22.5) |
| Race, No. (%) | ||
| Asian | 17 (2.6) | 34 (5.2) |
| Black or African American | 71 (10.8) | 130 (20.0) |
| Native American | 9 (1.4) | 9 (1.4) |
| Native Hawaiian/Pacific Islander | 2 (0.3) | 4 (0.6) |
| Multiracial | 54 (8.2) | 34 (5.2) |
| White | 489 (74.5) | 412 (63.5) |
| Othera | 11 (1.7) | 26 (4.0) |
| No response | 3 (0.5) | 0 (0.0) |
| Ethnicity, No. (%) | ||
| Latino/Hispanic | 136 (20.7) | 126 (19.4) |
| Non-Latino/non-Hispanic | 519 (79.1) | 523 (80.6) |
| No response | 1 (0.2) | 0 (0.0) |
| Gender, No. (%) | ||
| Female | 265 (40.4) | 360 (55.5) |
| Male | 362 (55.2) | 289 (44.5) |
| Other gender identityb | 28 (4.3) | 0 (0.0) |
| No response | 1 (0.1) | 0 (0.0) |
| Maternal education, No. (%) | ||
| <HS diploma | 149 (22.7) | 50 (7.5) |
| HS diploma or equivalent | 84 (12.8) | 139 (20.8) |
| Some college/associate degree | 160 (24.5) | 205 (30.7) |
| College degree | 166 (25.3) | 161 (24.1) |
| ≥Master’s degree | 97 (14.7) | 94 (14.1) |
| Not applicable | 0 (0.0) | 19 (2.8) |
Abbreviation: HS, high school.
Other race included biracial, Dominican, Hispanic, Mexican, and multiracial.
Other gender included nonbinary or gender queer, transgender, questioning, and prefer not to say.
Study 1
LLM scores strongly correlated with adolescents’ PME scores (Pearson r38 = 0.84; 95% CI, 0.71-0.91; P < .001), revealing that AI can reliably simulate human responses to text-only health messages.
Study 2
The LLM could estimate adolescent ratings based only on the text featured in ads (Pearson r202 = 0.53; 95% CI, 0.42-0.62; P < .001) (Figure). However, providing equivalent visual information to the LLM as was available to human raters significantly improved estimation (z score = 7.67; P < .001) and led to a strong correlation between LLM and adolescent ratings similar to study 1 (Pearson r202 = 0.83; 95% CI, 0.78-0.87; P < .001).
Figure. Correlations Between Adolescents’ Ratings of Message Effectiveness and GPT-4.0 (Open AI) Estimates Based on Text-Only or Text-Plus-Image Inputs.
Discussion
AI accurately simulated adolescents’ ratings of the effectiveness of vaping-prevention messages, in both text-only and text-plus-image formats. Importantly, the message databases were not available online, which means that the LLM generated novel ratings and did not merely retrieve ratings from memory. We appreciate the ethical issues surrounding AI and make no claims that LLMs generally make humanlike judgments or could entirely replace human research participants,6 and we acknowledge that research with additional samples and in other domains is warranted to corroborate the present findings.
Researchers and practitioners often face difficult decisions about which messages and images to use in health campaigns. The AI protocol developed here could facilitate pilot testing of a large number of text and image variations quickly and at very low cost and lead to meaningful improvements in the impact of health communications. The protocol could also offer insights about how to optimize messages about lifestyle (eg, diet, exercise), treatment options, vaccination, and other issues. AI thus has the potential to become a formidable tool for preventive medicine.
eAppendix 1. Human Participants and Methods in Study 1
eAppendix 2. Human Participants and Methods in Study 2
eAppendix 3. Protocol Used to Enable GPT-4.0 to Generate Effectiveness Ratings
eAppendix 4. Protocol Used to Describe Images in Study 2
eAppendix 5. Examples of Vaping-Prevention Messages Used in Study 1
eAppendix 6. Examples of Vaping-Prevention Ads Used in Study 2
eReferences
Data Sharing Statement.
References
- 1.Khera R, Butte AJ, Berkwits M, et al. AI in medicine—JAMA’s focus on clinical outcomes, patient-centered care, quality, and equity. JAMA. 2023;330(9):818-820. doi: 10.1001/jama.2023.15481 [DOI] [PubMed] [Google Scholar]
- 2.Walley SC, Wilson KM, Winickoff JP, Groner J. A public health crisis: electronic cigarettes, vape, and Juul. Pediatrics. 2019;143(6):e20182741. doi: 10.1542/peds.2018-2741 [DOI] [PubMed] [Google Scholar]
- 3.Rohde JA, Noar SM, Sheldon JM, Hall MG, Kieu T, Brewer NT. Identifying promising themes for adolescent vaping warnings: a national experiment. Nicotine Tob Res. 2022;24(9):1379-1385. doi: 10.1093/ntr/ntac093 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Noar SM, Gottfredson N, Vereen RN, et al. Development of the UNC perceived message effectiveness scale for youth. Tob Control. 2023;32(5):553-558. doi: 10.1136/tobaccocontrol-2021-056929 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Boynton MH, Sanzo N, Brothers W, et al. Perceived effectiveness of objective elements of vaping prevention messages among adolescents. Tob Control. 2023;32(e2):e228-e235. doi: 10.1136/tobaccocontrol-2021-057151 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Dillion D, Tandon N, Gu Y, Gray K. Can AI language models replace human participants? Trends Cogn Sci. 2023;27(7):597-600. doi: 10.1016/j.tics.2023.04.008 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
eAppendix 1. Human Participants and Methods in Study 1
eAppendix 2. Human Participants and Methods in Study 2
eAppendix 3. Protocol Used to Enable GPT-4.0 to Generate Effectiveness Ratings
eAppendix 4. Protocol Used to Describe Images in Study 2
eAppendix 5. Examples of Vaping-Prevention Messages Used in Study 1
eAppendix 6. Examples of Vaping-Prevention Ads Used in Study 2
eReferences
Data Sharing Statement.
