Abstract
Objectives
Artificial intelligence (AI) can be used to measure sentiment and reading level, facilitating its use to improve nutrition communication, though AI has not yet been tested for such analysis of nutrition articles. The present study investigated this by looking at reading level and sentiment in magazines aimed at various demographics as determined with ChatGPT.
Materials and methods
Sentiment analysis was conducted on a sample of articles about nutrition from legacy media collected over 1 year. Reading level of the articles was determined using the Simple Measure of Gobbledygook (SMOG) formula. Nutrition content sub-themes were examined for reading level and sentiment.
Results
Average sentiment ratings remained close to neutral (mean = 0.26 ± 2.05), with no consistent changes over time. Articles written at higher reading levels showed more positive sentiment values than those at lower reading levels. Of the twelve nutrition sub-themes were identified, the most prevalent were food/diet recommendations for health (23.8%), articles about celebrity diets (15.6%), cooking and culinary articles (14.1%), and food insecurity (10.7%). Comparison of sub-samples of sentiment ratings or Simple Measure of Gobbledygook (SMOG) scores done by a human coder versus ChatGPT were not different (P = NS).
Discussion
Reading levels are relatively high overall. These varied by source periodical and theme as would be expected with editors targeting varied readership groups. Analysis by nutrition themes showed differences in sentiment that were concurrent with the type of content analyzed.
Conclusion
This study demonstrated that AI such as ChatGPT can be utilized as a for assessment of sentiment and reading levels in nutrition articles.
Keywords: AI, sentiment analysis, SMOG, nutrition, diet, legacy media
Background and significance
Legacy media are media that existed prior to the establishment of the internet. These included print media such as newspapers and magazines. Gaba and Bennett1 collected material from a sample of legacy media, and developed a data base of articles, editorials, and advertising related to public health themes. One of these themes was food and nutrition. The data used in the present study was derived from articles coded as including food and nutrition topics, extracted from this larger data set.
Legacy media have a range of reading levels, depending upon their target demographics. Having an understanding of how nutrition content is presented, and at what reading level it is composed would be, along with sentiments expressed, an important factor in understanding how this information is transmitted.
According to US Literacy Statistics, the literacy rate in the United States is 79%. According to this source, 54% of U.S. adults demonstrate literacy skills below a sixth-grade level, with 20% below a fifth-grade level.2 The Literacy Project reports that over half of American adults read below a sixth grade level,3 so it may be challenging for many people to understand the information they read.
Other considerations besides reading level in evaluating health communications have been identified. These include graphics, layout/typography, learning stimulation, and cultural appropriateness.4 Lowering the reading level alone may not be enough to produce desired outcomes. Improvement in readability is not beneficial if created at the expense of acceptability and trust of the writers.5 It is not just what is said, but the tone of how it is said. Consumers of information may have unique perspectives that interact with readability and tone to affect trust. The present study investigated this concept by looking at reading level along with sentiment and source magazines aimed at various demographics.
Sentiment analysis is a means to identify and evaluate feeling or emotional charge in text-based media, or transcripts of audio or video material. It evaluates content in a consistent and systematic way.6 Through targeted content analysis, material can be identified as positive, negative, or neutral.7 It can also be used to calculate magnitude of strong or weak emotions.8 The most commonly used model in emotion recognition research is that of Paul Ekman, which involves 6 basic emotions: happiness, sadness, anger, fear, surprise, and disgust.9
To conduct a large-scale sentiment analysis, these basic terms are expanded into a lexicon or dictionary of words identified as having specific values. Many lexicons have been developed for this purpose, including Subjectivity Wordlist, WordNet-Affect, and SentiWordNet.10
AI has been used to facilitate sentiment analysis of text in multiple contexts. Some examples are Fatouros et al.,11 Bal et al.,12 and Lu et al.13 Some specific examples where this has been applied to food and nutrition content include Sauraa et al.14 and Molenaar et al.6
The objectives of the current study were:
To evaluate the reading level by Simple Measure of Gobbledygook (SMOG) score of a sample of nutrition articles from legacy media utilizing ChatGPT.
To evaluate the sentiment of a sample of nutrition articles using ChatGPT.
To validate the accuracy of both of those measures by comparing a 10% sub-sample to the numbers generated for that same sub-sample by a human coder.
Compare nutrition themes derived by content analysis of the articles, across sentiment and reading levels.
The overall goal of this study was to see if data extraction by ChatGPT from a sample of nutrition articles was valid and useful in evaluating these articles and providing results that were useful.
Methods
Legacy media data set as a source for nutrition articles
The data used in the present study was derived from articles coded as including food and nutrition topics, extracted from a larger data set.1 Duplicate articles and advertising were removed prior to further analysis.
The total number of articles from all sources was 475. The largest number of articles (n = 89) appeared in Time magazine, with Scientific American a close second with n = 81. The largest number of articles were published in the first quarter of the year, with the largest number appearing in April issues.
Sentiment analysis methods
Sentiment analysis is based on a lexicon or dictionary of words identified as having specific values. Rao et al.8 presented algorithms for constructing a word-level and topic-level emotional dictionaries. In the dictionary, each word is associated with the distribution on a series of human emotions, and assigned a number that is either positive or negative, with a zero score being neutral. Kumar et al.15 provide an extensive review of these methods as they have evolved in recent years and describe available lexicons in English.
In this study, ChatGPT has been prompted to assess the sentiments expressed in each of the 475 nutrition articles evaluated. Specific prompts used are shown in Appendix SA. Terms generated during this process were categorized as positive, neutral or negative in tone. For example, words such as “enthusiastic,” “hopeful,” or “inspirational” were rated as positive, whereas “urgent,” “cautionary,” and “skeptical” were rated as more negative in sentiment.
Evaluation of reading level
Readability was assessed using the SMOG formula, a widely used method that estimates the years of education required to fully comprehend a text based on the frequency of polysyllabic words.16 The SMOG scale has a long history of use as a reliable method to evaluate the reading level of text in any medium. We chose this method for its ease of use by both AI and human evaluators.
Validation of AI generated results
Methods of sentiment analysis and the importance of validation of the results obtained are described in detail by van Atteveldt et al.17 To validate the accuracy of data extraction and classification by ChatGPT, random subsets of data equal to 10% (n = 47) of the total data set were compared for the same functions performed by a human coder. Documents were selected based on a list of 47 random numbers generated using an online random number generator corresponding to their numbers in the data file.18 This is comparable to a study by Lappeman et al. that described assessment of inter-rater reliability to validate the results of sentiment analysis by an AI.19 Their sample for topic analysis was equal to 9.26% of their total sample, with an average margin of error of 0.26%.
Since the specific lexicon used in ChatGPT to perform sentiment analysis is not known, to verify sentiment analysis, the hand-analysis used sentiment terms from a publicly available word lexicon. For each document in the sub-set, a human evaluator tallied all sentiment-related words in the first, third, and last paragraphs with rating numbers from the AFINN-111 lexicon.20 These were then averaged to provide an overall sentiment score for each document.
To verify SMOG reading level analysis, the SMOG was calculated by hand for each document in the sub-sample using the formula: square root of the number of words with more than 3 syllables in thirty sentences selected from the beginning, middle, and end of a document.16
Development of sub-themes
Following the analysis of sentiment and reading level, the 475 articles were reexamined to identify further sub-themes within these nutrition articles. Each article was read, and key words from the title and body of each text were extracted. These words were listed for all articles, and then similar topics were consolidated into theme codes, for example, “agriculture,” “farming,” and “growing crops” were all combined into 1 code. This process resulted in twelve distinct sub-themes, that were applied to each of the articles so that thematically similar articles could be further analyzed. Comparisons by sub-theme were done by source periodical, month of publication, SMOG reading level, and AI determined sentiment score.
Ethics statement
The study examined publicly available printed materials. Data was extracted from these printed materials for analysis, but no copyrighted material was incorporated into the final study. The research was conducted exclusively by the authors; no other people were involved. Since there were no human subjects involved, and no consents or permissions were required, the study was determined to be exempt from IRB ethical review by the human subjects review board at the first author’s institution.
Results
Comparisons of ChatGPT versus a human investigator for sentiment analysis and SMOG scores in a randomly selected sub-sample of the total data set were compared using paired-samples t-tests, with 46 degrees of freedom. The critical value for t = ± 2.014 for a statistically significant difference. For sentiment scores, the mean score calculated by AI was 0.00, with the mean calculated by a human investigator equal to 0.70. [t = −.73], 1-tailed P = .234 (NS), 2-tailed P = .467 (NS). The 95% CI for this difference was [−2.63,1.22]. For SMOG scores, the mean calculated by AI was 11.7 (range 8-16), and by the human investigator was 11.2 (range 8-17). [t = 1.24] 1-tailed P = .111 (NS), 2-tailed P = .222 (NS). The 95% CI for this difference was [−.35,1.45]. The 95% CI for each of these measures includes 0, indicating there is not statistically significant difference. Although the ranges were different, the mean values for both the sentiment scores and SMOG scores were not statistically different. This provides a measure of validity for the ChatGPT scores generated for the total data set as reported below.
The remaining analysis was conducted on the entire sample (N = 475). Comparisons were done utilizing SMOG reading level scores, sentiment analysis polarities, and categorization of articles by nutrition sub-themes. The results of each of these will be discussed in turn below.
Analysis of SMOG reading level scores
SMOG statistics had a range from 8 (eigth grade reading level) to 17 (first year graduate student), with a mean of 11.3 (high school junior). The standard deviation was 2.2, with a variance of 4.8.
Mean scores and variability of SMOG reading level across the 11 magazines studied are shown in Figure 1.
Figure 1.

Mean SMOG grade by source publication. This figure shows the mean and variability of SMOG reading level across the 11 magazines studied. The average SMOG grade across all publications was 11.34 (SD ± 2.21). Most magazines clustered around this average, with Scientific American reporting the highest mean readability level (12.16) and the Advocate the lowest (10.33).
We further compared SMOG scores across magazines using a Kruskal-Wallis test. When significant, we conducted pairwise Mann-Whitney U tests with Holm adjustment. Effect sizes for pairwise comparisons are reported as Cliff’s delta with 95% CIs.
Scientific American, having the highest average SMOG score, was significantly different from Men’s Journal, Vogue, Oprah and Elle. Comparisons that were significant initially, but were not statistically significant after adjustment are shown below, noted as NS. These results are shown in Table 1.
Table 1.
Comparisons of SMOG scores between periodicals.
| Comparison | P for adjusted U test | Effect size | 95% CI |
|---|---|---|---|
| Scientific American—Men’s Journal | .0000156 | −0.399 | [−0.542, −0.233] |
| Scientific American—Vouge | .000616 | 0.391 | [0.210, 0.547] |
| Scientific American—Oprah | .003 | −0.371 | [−0.523, −0.197] |
| Scientific American—Elle | .013 | −0.434 | [−0.617, −0.206] |
| Scientific American—Essence | .122 (NS) | −0.342 | [−0.529, −0.125] |
| MS—Men’s Journal | .152 (NS) | −0.405 | [−0.639, −0.103] |
| MS—Vogue | .264 (NS) | 0.401 | [0.0839, 0.645] |
| MS—Elle | .265 (NS) | −0.461 | [−0.691, −0.148] |
| MS—Oprah | .413 (NS) | 0.386 | [0.124, 0.598] |
| The Advocate—Men’s Journal | .653 (NS) | 0.493 | [0.128, 0.740] |
Analysis of sentiment
The most frequently generated sentiment terms included “serious,” “cautionary,” and “inspirational,” suggesting that nutrition content in legacy media was often framed in a serious and preventive tone. Average sentiment values ranged from −0.33 to 1.00, with most clustering around neutral (0.0). Averages by periodical were: the Advocate (−0.33), Elle (0.65), Essence (0.95), Gentlemen’s Quarterly (GQ) (0.46), Men’s Journal (0.19), MS (−0.33), Oprah (0.16), Out (1.00), Scientific American (0.89), Time (−0.35), and Vogue (0.03). Between periodical differences were not statistically significant.
The average sentiment varied by month throughout the year, with a range of +0.92 in February to −0.19 in December. The trend line for this data is shown in Figure 2.
Figure 2.

Average ChatGPT generated sentiment score by month. This figure shows the trend line for average sentiment score by month. The overall mean sentiment score was near neutral (0.26), but variable across the range of sentiments (SD ± 2.05).
Interaction of SMOG and sentiment scores
To consider the interaction of reading grade level on sentiment tone, sentiment polarity values were compared by SMOG scores. Articles written at higher reading levels (college or graduate level) had the most positive polarity values (0.49-0.67), while those at lower grade levels showed more neutral to negative values although there was not a clear trend among these grade levels.
Comparison of nutrition sub-themes to SMOG and sentiment
In order to further interpret the results found above, similar articles were grouped according to twelve nutrition sub-themes. Articles grouped under each of these themes were evaluated by source periodical, month of publication, SMOG reading level, and AI determined sentiment score.
Descriptive statistics with nutrition sub-themes
Table 2 shows the frequency of each of the twelve nutrition sub-themes within the source periodicals. Four sub-themes accounted for the majority of articles. These were cooking and culinary articles, articles about celebrity’s diets and food preferences, food/diet recommendations for health, and food insecurity. Food and immunity was broken out as a separate category under diet and health, but did not add substantially to the large number of articles in this sub-theme.
Table 2.
Frequency of nutrition themes by source periodical.
| Nutrition themes | The Advocate | Elle | Essence | GQ | Men’s Journal | MS | Oprah | Out | Scientific American | Time | Vogue | Totals (%) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Agriculture and farms | 0 | 0 | 0 | 1 | 2 | 5 | 0 | 0 | 7 | 11 | 4 | 30 (6.3) |
| Animals/meat | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 6 | 8 | 0 | 14 (2.9) |
| Cooking/culinary | 0 | 5 | 3 | 4 | 9 | 0 | 7 | 1 | 5 | 15 | 18 | 67 (14.1) |
| Celebrity diet | 0 | 6 | 9 | 32 | 10 | 0 | 6 | 2 | 0 | 1 | 8 | 74 (15.6) |
| Diet & health promotion | 0 | 2 | 2 | 11 | 37 | 0 | 10 | 1 | 36 | 4 | 10 | 113 (23.8) |
| Eating disorders | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 2 | 2 | 6 (1.3) |
| Food insecurity | 0 | 3 | 1 | 0 | 1 | 8 | 3 | 0 | 4 | 26 | 3 | 51 (10.7) |
| Food policies and programs | 0 | 3 | 0 | 3 | 0 | 2 | 2 | 0 | 7 | 15 | 5 | 37 (7.8) |
| Food and immunity | 2 | 0 | 3 | 2 | 2 | 1 | 0 | 0 | 1 | 0 | 2 | 11 (2.3) |
| Restaurants | 0 | 0 | 0 | 5 | 3 | 0 | 1 | 0 | 0 | 3 | 14 | 28 (5.9) |
| Diet, stress, and mental health | 0 | 0 | 3 | 1 | 0 | 0 | 1 | 2 | 3 | 4 | 3 | 15 (3.2) |
| Weight loss | 1 | 0 | 1 | 1 | 10 | 2 | 1 | 0 | 11 | 0 | 2 | 29 (6.1) |
| Totals (%) | 3 (0.6) | 20 (4.2) | 22 (4.6) | 60 (12.6) | 74 (15.6) | 18 (3.8) | 31 (6.5) | 6 (1.3) | 81 (17.1) | 89 (18.7) | 74 (14.9) | 475 (100) |
Looking at these frequencies by month of publication, in Table 3, the highest frequencies for the 4 most common sub-themes are all the highest early in the year, with the exception of celebrity diets, with was the highest in November.
Table 3.
Frequency of nutrition themes by month.
| Nutrition themes | January | February | March | April | May | June | July | August | September | October | November | December | Totals (%) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Agriculture and farms | 4 | 3 | 3 | 2 | 3 | 2 | 2 | 3 | 1 | 1 | 3 | 3 | 30 (6.3) |
| Animals/meat | 1 | 1 | 2 | 3 | 3 | 2 | 0 | 1 | 0 | 0 | 1 | 0 | 14 (2.9) |
| Cooking/culinary | 11 | 6 | 6 | 15 | 8 | 3 | 6 | 2 | 2 | 2 | 2 | 4 | 67 (14.1) |
| Celebrity diet | 7 | 8 | 5 | 7 | 5 | 3 | 4 | 5 | 5 | 8 | 10 | 7 | 74 (15.6) |
| Diet and health promotion | 24 | 19 | 12 | 14 | 4 | 8 | 5 | 10 | 4 | 4 | 5 | 4 | 113 (23.8) |
| Eating disorders | 2 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 6 (1.3) |
| Food insecurity | 0 | 2 | 6 | 11 | 9 | 3 | 1 | 3 | 4 | 3 | 2 | 7 | 51 (10.7) |
| Food policies and programs | 2 | 8 | 2 | 5 | 5 | 4 | 2 | 2 | 0 | 4 | 2 | 1 | 37 (7.8) |
| Food and immunity | 0 | 0 | 2 | 1 | 1 | 2 | 2 | 0 | 1 | 0 | 2 | 0 | 11 (2.3) |
| Restaurants | 0 | 0 | 1 | 6 | 6 | 3 | 1 | 3 | 3 | 0 | 2 | 3 | 28 (5.9) |
| Diet, stress, and mental health | 1 | 0 | 1 | 1 | 1 | 3 | 1 | 1 | 0 | 1 | 5 | 0 | 15 (3.2) |
| Weight loss | 6 | 3 | 2 | 1 | 3 | 0 | 4 | 3 | 3 | 1 | 1 | 2 | 29 (6.1) |
| Totals (%) | 58 (12.2) | 50 (10.5) | 43 (9.1) | 67 (14.1) | 48 (10.1) | 33 (6.9) | 29 (6.1) | 33 (6.9) | 23 (4.8) | 24 (5.1) | 36 (7.6) | 31 (6.5) | 475 (100) |
Table 4 shows the mean and range of SMOG scores and sentiment ratings within each nutrition sub-theme. The theme with the lowest average SMOG score was agriculture and farms. The highest average SMOG score was in the category of weight loss. This may have been skewed upward by many articles in Scientific American about the biology of weight loss and gain. Sentiment ratings of these nutrition articles were more frequently positive than negative. The 42.1% had ratings less than 0, with 57.9% having positive sentiment ratings. The overall mean of sentiment ratings was 0.27 with standard deviation of 2.055.
Table 4.
Nutrition themes by SMOG reading level and sentiment scores.
| Nutrition themes | SMOG grade level—mean (range) | Sentiment evaluation—means (range = −3 to +3) |
|---|---|---|
| Agriculture and farms | 10.03 (8-16) | +0.30 |
| Animals/meat | 11.57 (9-14) | +0.40 |
| Cooking/culinary | 11.32 (8-17) | −0.15 |
| Celebrity diet | 10.97 (8–16) | +0.34 |
| Diet and health promotion | 11.42 (7-16) | +0.43 |
| Eating disorders | 10.33 (9-13) | +0.33 |
| Food insecurity | 11.29 (8-16) | −0.33 |
| Food policies and programs | 11.03 (7-15) | +0.41 |
| Food and immunity | 11.18 (8-15) | +0.54 |
| Restaurants | 11.39 (8-15) | −0.03 |
| Diet, stress, and mental health | 10.80 (8-15) | +0.73 |
| Weight loss | 12.58 (9-17) | +0.24 |
It was not surprising that food insecurity had the most negative sentiment rating, but cooking and culinary articles were also slightly more negative than positive. The most positive overall sentiment was in articles about diet, stress, and mental health, perhaps because they expressed hope that food could address these issues.
Figure 3 is a heat map of the distribution of sentiment terms across reading grade levels. Articles at higher reading levels (college and above) were more frequently coded as “serious,” “cautionary,” and “hopeful,” while lower-level texts showed a more mixed distribution, suggesting that more complex articles tend to have a more formal or preventive tone.
Figure 3.

Sentiment frequency by SMOG reading level. This figure is a heat map showing the relationship between sentiment frequency and reading level. Higher SMOG grade level reading score is consistent with a more serious sentiment.
Discussion
The outcomes of this study have several implications for health and nutrition communications and public health messaging. Firstly, it has provided an example of the feasibility of using AI to examine reading level and sentiment in a sample of nutrition articles extracted from legacy media. This overall process provides a modifiable measure of readability and tone of these communications.
The emotional tone or sentiment of communications can have a significant impact on the degree to which health messages gain acceptance by readers. In examining health communications via Tweets, Jenkins at al.21 found that content that conveyed positive emotions such as bravery or success enhanced interpretation of the message’s authenticity and trustworthiness. The contrast in tone of messaging between academic publications and lay print media was examined by Scheper et al.22 They found that as 1 might expect, their sample of academic literature was largely neutral in tone and specific in content, whereas the lay print media sample was heavily optimistic in tone but with less specificity in content. These factors should be taken into consideration when developing health communications to maximize clarity and promote trust for readers.
Our comparisons of articles across AI-generated sentiment values by human-identified nutrition themes showed consistent and subjectively appropriate sentiments applied to articles categorized under each theme. Articles about food insecurity had the most negative sentiment scores, with cooking and culinary information also having a more negative sentiment. The latter may have reflected an increased degree of difficulty in cooking during the pandemic lockdowns during 2020. Articles about diet for stress and mental health were rated the most positively, along with articles about food and immunity. These may have been written expressing more positive sentiment to encourage readers to adopt better eating habits.
A second area with implications for health communications is literacy and reading level. The relatively low levels of reading ability identified for US adults are in sharp contrast to evaluations of science communications, such as the Cochrane Plain Language Summaries (PLSs). The purpose of these is to provide clear and easily understood summaries of these scientific reviews for the general public. An evaluation of SMOG reading scores in Cochrane PLSs found an average of 14.9, that is, equivalent to a third-year college undergraduate.23 Furthermore, this same study found that PLSs had a primarily negative emotional tone, such as anxiety, sadness, and hostility. In addition to the Cochrane PLSs, from medical and non-medical organizations were found to have even higher reading scores, with non-medical summaries averaging a SMOG score of 15.23, and summaries from medical literature averaging 15.51. While this difference was statistically significant, more importantly it shows that “lay summaries” in both medical and non-medical literature is written at the reading level of a college senior.24
Our results showed that our sample of nutrition articles from legacy media had average reading levels somewhere in between assessments based on social media and Cochrane reviews, with an average SMOG reading level of 11.3. In sentiment analysis, these nutrition articles were overall neutral in sentiment, including most frequently serious, cautionary, and hopeful tones. This may reflect somewhat more positive attitudes toward nutrition content, versus the varied medical topics listed in Cochrane reviews.
Comparison of the SMOG scores by U test showed significant differences between Scientific American compared with Men’s Journal, Vogue, Oprah, and Elle. Other comparisons were not statistically significant. These results are not surprising since Scientific American had the highest average SMOG score, with relatively little variability in average scores across all periodicals (mean 11.34 with SD ± 2.21).
As with any research, this study had strengths and weaknesses. Strengths of this study include utilization of periodicals collected during the first 12 months of the during the COVID-19 pandemic. Specific nutrition issues occurring during that time are well documented and could be aligned to our findings as described above. Multiple periodicals were included with different target readership groups, allowing for cross-readership comparisons.
Use of AI in data calculation was shown to be effective, and comparable to a human carrying out the same functions. This study demonstrated that AI such as ChatGPT can be utilized as a practical tool for simple assessment of sentiment and calculation of reading levels in a sample of documents. To the best of our knowledge, our paper is the first of its kind to utilize ChatGPT as a data extraction tool for sentiment analysis of legacy media. This novel application may have further uses in research.
Weaknesses are that the convenience data set may not be representative of all legacy media in print at this time. Also, they were all in English, which may have missed important messaging issues for other language readers. Since sentiment analysis is an area with global applications, lexicons have been created in multiple languages besides English, including French25; Spanish26,27; Hindi28; Chinese29; Portuguese30; Japanese31; Bangla32; Turkish33; and Italian.34 Further research in this area including multi-lingual and cross-lingual sentiment analysis may prove beneficial in optimizing health communications for global readership.
Another issue not addressed in this study is the capacity of AI to edit text to manipulate the sentiment and reading level of an article to make it more accessible for target populations. For example, a recent study by Nasser et al. demonstrated that ChatGPT-4o could improve readability of patient education materials on orthopedic surgery by converting these from an average SMOG reading score of 10.2 to an average of 6.6 without changing the primary information conveyed.35 This type of AI-generated change may also be useful for aligning reading levels of nutrition education materials to a range of clients with minimal effort by providers. Changes to reading levels represent the simpler applications of this technology. The potential impacts of modifying communications based on gradients of sentiment have yet to be explored.
Conclusions
Mistargeting of communications can lead to rejection of these important messages. Ensuring that information is appropriately accessible is not only a matter of style but of public health equity. Trust in communications can be enhanced when readers feel that information is relevant to them and neither confusing nor patronizing. Messaging also needs to take into consideration the emotional charge attached to health topics. Public health communications need to be written for a range of readers to accommodate both needs for emotional tone of information, and comprehension at appropriate reading levels. AI can facilitate analysis of these as has been demonstrated in the examples of legacy media evaluated here.
Supplementary Material
Contributor Information
Ann Gaba, Department of Environmental, Occupational, and Geospatial Health Sciences, Graduate School of Public Health and Health Policy, City University of New York, New York, NY 10027, United States.
Safa Amir, Department of Epidemiology and Biostatistics, Graduate School of Public Health and Health Policy, City University of New York, New York, NY 10027, United States.
Haley Liebman, Department of Environmental, Occupational, and Geospatial Health Sciences, Graduate School of Public Health and Health Policy, City University of New York, New York, NY 10027, United States.
Nevin Cohen, Department of Health Policy and Management, Graduate School of Public Health and Health Policy, City University of New York, New York, NY 10027, United States.
Steven Cordova, Department of Natural Sciences, Health and Wellness, Miami Dade College, Miami, FL 33132, United States; Department of Medical Education, Hospital Auxilio Mutuo, San Juan, 00919, Puerto Rico.
Sergio Costa, College of Distance Education, U.S. Naval War College, Newport, RI 02841, United States.
Ashish Joshi, School of Public Health, University of Memphis, Memphis, TN 30152, United States.
Author contributions
Ann Gaba (Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Supervision, Visualization, Writing—original draft, Writing—review & editing), Safa Amir (Formal analysis, Investigation, Methodology, Visualization, Writing—original draft, Writing—review & editing), Haley Liebman (Conceptualization, Methodology), Nevin Cohen (Conceptualization, Methodology, Writing—review & editing), Steven Cordova (Writing—review & editing), Sergio Costa (Writing—review & editing), and Ashish Joshi (Writing—review & editing)
Supplementary material
Supplementary material is available at JAMIA Open online.
Funding
This work was supported by PSC-CUNY award TRADB-55-132 #67432-00 55 to the first author.
Conflicts of interest
The authors declare no conflicts of interest. The funder had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.
Data availability
Data used in this study is available from the corresponding author upon reasonable request.
Institutional review board statement
This research was determined to be exempt from IRB review since neither human or animal participants nor sensitive personal information was involved in its conduct.
Statement of use regarding generative AI
ChatGPT was used to calculate SMOG reading scores, and to perform sentiment analysis on the documents under study. All coding, data analysis, drafting and editing of the manuscript, and production of tables, graphs, and figures are entirely the original work of the authors. Prompts used are shown in Appendix SA.
References
- 1. Gaba A, Bennett R. Comparing public health related material in print and web page versions of legacy media. JAMIA. 2024;7:ooae104. 10.1093/jamiaopen/ooae104 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. US Literacy Statistics. Accessed June 21, 2025. https://www.magnetaba.com/blog/us-literacy-statistics
- 3. The Literacy Project. Accessed May 25, 2025. https://www.sparxservices.org/blog/us-literacy-statistics-literacy-rate-average-reading-level
- 4. Finnie RK, Felder TM, Linder SK, Mullen PD. Beyond reading level: a systematic review of the suitability of cancer education print and web-based materials. J Cancer Educ. 2010;25:497-505. 10.1007/s13187-010-0075-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Mac O, Ayre J, McCaffery K, Boroumand F, Bell K, Muscat DM. The readability study: a randomized trial of health information written at different grade reading levels. J Gen Intern Med. 2025;40:1820-1828. 10.1007/s11606-024-09200-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Molenaar A, Jenkins EL, Brennan L, Lukose D, McCaffrey TA. The use of sentiment and emotion analysis and data science to assess the language of nutrition-, food- and cooking-related content on social media: a systematic scoping review. Nutr Res Rev. 2024;37:43-78. 10.1017/S0954422423000069 [DOI] [PubMed] [Google Scholar]
- 7. Nandwani P, Verma R. A review on sentiment analysis and emotion detection from text. Soc Netw Anal Min. 2021;11:81. 10.1007/s13278-021-00776-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Cui J, Wang Z, Ho SB, Cambria E. Survey on sentiment analysis: evolution of research methods and topics. Artif Intell Rev. 2023;6:1-42. 10.1007/s10462-022-10386-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Alswaidan N, Mohamed El Bachir M. A survey of state-of-the-art approaches for emotion recognition in text. Knowl Inf Syst. 2020;62:2937-2987. 10.1007/s10115-020-01449-0 [DOI] [Google Scholar]
- 10. Rao Y, Lei J, Wenyin L, et al. Building emotional dictionary for sentiment analysis of online news. World Wide Web. 2014;17:723-742. 10.1007/s11280-013-0221-9 [DOI] [Google Scholar]
- 11. Fatouros G, Soldatos J, Kouroumali K, Makridis G, Kyriazis D. Transforming sentiment analysis in the financial domain with ChatGPT. MLWA. 2023;14:100508. 10.1016/j.mlwa.2023.100508 [DOI] [Google Scholar]
- 12. Bal M, Kara Aydemir AG, Coşkun M. Exploring YouTube content creators’ perspectives on generative AI in language learning: insights through opinion mining and sentiment analysis. PLoS One. 2024;19:e0308096. 10.1371/journal.pone.0308096 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Lu L, Zhu Y, Yang J, et al. Healthcare professionals and the public sentiment analysis of ChatGPT in clinical practice. Sci Rep. 2025;15:1223. 10.1038/s41598-024-84512-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Saura JR, Reyes-Menendez A, Thomas SB. Gaining a deeper understanding of nutrition using social networks and user generated content. Internet Interv. 2020;20:100312. 10.1016/j.invent.2020.100312 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Kumar M, Khan L, Chang H-T. Evolving techniques in sentiment analysis: a comprehensive review. PeerJ Comput Sci. 2025;11:e2592 10.7717/peerj-cs.2592 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. McLaughlin GH. SMOG grading-a new readability formula. J Read. 1969;5:639-646. https://ogg.osu.edu/media/documents/health_lit/WRRSMOG_Readability_Formula_G._Harry_McLaughlin__1969_.pdf [Google Scholar]
- 17. van Atteveldt W, van der Velden M, Boukes M. The validity of sentiment analysis: comparing manual annotation, crowd-coding, dictionary approaches, and machine learning algorithms. Commun Methods Meas. 2021;15:121-140. 10.1080/19312458.2020.1869198 [DOI] [Google Scholar]
- 18. Random Number Generator. Accessed June 3, 2025. https://www.random.org/sequences/
- 19. Lappeman J, Clark R, Evans J, et al. Studying social media sentiment using human validated analysis. MethodsX. 2020;7:100867. 10.1016/j.mex.2020.100867 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Nielsen FA. A new ANEW: evaluation of a word list for sentiment analysis in microblogs. In: Proceedings of the ESWC2011 Workshop on ‘Making Sense of Microposts: Big Things Come in Small Packages’. Heraklion, Crete. 2011:93-98. https://www2.imm.dtu.dk/pubdb/edoc/imm6006.pdf
- 21. Jenkins EL, Ilicic J, Molenaar A, Chin S, McCaffrey TA. Strategies to improve health communication: can health professionals be heroes? Nutrients. 2020;12:1861. 10.3390/nu12061861 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Scheper A, Rosenfeld C, Dubljević V. The public impact of academic and print media portrayals of TMS: shining a spotlight on discrepancies in the literature. BMC Med Ethics. 2022;23:25. 10.1186/s12910-022-00760-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Banić A, Fidahić M, Šuto J, et al. Conclusiveness, linguistic characteristics and readability of Cochrane plain language summaries of intervention reviews: a cross-sectional study. BMC Med Res Methodol. 2022;22:240. 10.1186/s12874-022-01721-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Bralić N, Mijatović A, Marušić A, et al. Conclusiveness, readability and textual characteristics of plain language summaries from medical and non-medical organizations: a cross-sectional study. Sci Rep. 2024;14:6016. 10.1038/s41598-024-56727-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Duval D, Petry F. Automated analysis of the media tone: construction and use of the French version of the Lexicoder Sentiment Dictionary. Can J Polit Sci. 2016;49:197-220. 10.1017/S000842391600055X [DOI] [Google Scholar]
- 26. Jiménez-Zafra SM, Taulé M, Martín-Valdivia MT, Ureña-López LA, Martí MA. SFU review SP -NEG: a Spanish corpus annotated with negation for sentiment analysis. A typology of negation patterns. Lang Resources Eval. 2018;52:533-569. 10.1007/s10579-017-9391-x [DOI] [Google Scholar]
- 27. Navas-Loro M, Rodríguez-Doncel V. Spanish corpora for sentiment analysis: a survey. Lang Resources Eval. 2020;54:303-340. 10.1007/s10579-019-09470-8 [DOI] [Google Scholar]
- 28. Jha V, Savitha R, Shenoy PD, Venugopal KR, Sangaiah AK. A novel sentiment aware dictionary for multi-domain sentiment classification. Comput Electr Eng. 2018;7:585-597. 10.1016/j.compeleceng.2017.10.015 [DOI] [Google Scholar]
- 29. Xu G, Yu Z, Yao H, Li F, Meng Y, Wu X. Chinese text sentiment analysis based on extended sentiment dictionary. IEEE Access. 2019;7:43749-43762. 10.1109/ACCESS.2019.2907772 [DOI] [Google Scholar]
- 30. Pereira DA. A survey of sentiment analysis in the Portuguese language. Artif Intell Rev. 2021;54:1087-1115. 10.1007/s10462-020-09870-1 [DOI] [Google Scholar]
- 31. Masataka N, Chihiro H, Hisae K, et al. Japanese Dictionary for Sentiment Analysis of Counselling Text. In: Proceedings of the 9th International Conference on Human-Agent Interaction. Virtually and Nagoya, Japan. 2021:311-315. 10.1145/3472307.3484663 [DOI]
- 32. Bhowmik NR, Arifuzzaman M, Mondal MRH. Sentiment analysis on Bangla text using extended lexicon dictionary and deep learning algorithms. Array (New York). 2022;13:100123. 10.1016/j.array.2021.100123 [DOI] [Google Scholar]
- 33. Alkan BB, Karakus L, Direkci B. Generating a sentiment dictionary in R and dictionary-based sentiment analysis in Turkish texts. Digit Scholarsh Humanit. 2023;38:461-476. 10.1093/llc/fqac093 [DOI] [Google Scholar]
- 34. Martinis MC, Zucco C, Cannataro M, Tsitouras C, Simos T. VADER-IT: a sentiment analysis tool for the Italian language. AIP Conf Proc. 2024;3094. 10.1063/5.0210834 [DOI]
- 35. Nassar J, Farias M, Ammar L, et al. Bridging health literacy gaps in spine care: using ChatGPT-4o to improve patient-education materials. J Bone Joint Surg Am. 2025;107:2131-2140. 10.2106/JBJS.24.01484 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
Data used in this study is available from the corresponding author upon reasonable request.
