Analysis of Potential Associations of JUUL Flavors with Health Symptoms Based on User-Generated Data from Reddit

Abstract

Background:

The rise of the popular e-cigarette, JUUL, has been partly attributed to various teen-friendly e-liquid flavors offered. However, the possible health risks associated with each e-liquid flavor still remain unclear. This research focuses on the possible associations between JUUL flavors and health symptoms using social media data from Reddit.

Methods:

Keyword filtering was used to obtain 5,746 JUUL flavor-related posts and 7,927 health symptom-related posts from June 2015 to April 2019 from Reddit. Posts from September 2016 to April 2019 were used to conduct temporal analysis for 9 health symptom categories and the 8 JUUL flavors. Finally, associations between the JUUL flavors and health symptom categories were examined on the user level using generalized estimating equation (GEE) models.

Results:

According to our temporal analysis, Mango and Mint were the most discussed JUUL flavors on Reddit. Respiratory and throat symptoms were the most discussed health issues together with JUUL on Reddit over time. Respiratory symptoms had potential associations with the Mango, Mint, and Fruit JUUL flavors. Digestive symptoms had a potential association with the Crème flavor, psychological symptoms had a potential association with the Cucumber flavor, and cardiovascular symptoms had a potential association with the tobacco flavors.

Conclusions:

Mango and Mint were the two most mentioned JUUL flavors on Reddit. Certain JUUL flavors were more likely to be mentioned together with certain categories of health symptoms by the same Reddit users. Our findings could prompt further medical research into the health symptoms associated with different e-liquid flavors.

Introduction

E-cigarette development was promoted in the early 2000s as an effective tool for smoking cessation. However, in recent years, there has been an alarming amount of e-cigarette use, especially among youth in the United States who have never used traditional cigarettes.¹ Flavor variety of e-liquid, the main component in e-cigarettes, has been cited as one of the main reasons for the increase in e-cigarette use.² E-liquid comes in over 7,000 flavors, which attract users because they each provide unique experiences.³ Although proponents of flavorings say that they help adult smokers quit,⁴ e-liquid flavors often contain flavor chemicals that have potential risks to users. Many of the chemicals have been labelled as generally safe for ingestion by the Flavor Extracts Manufacturers Association (FEMA),⁵ but less is known about their health effects when heated and inhaled by e-cigarette users. Recent studies have focused on discovering which flavors and flavor chemicals are cytotoxic when inhaled through e-cigarette aerosol. A study found that menthol, coffee, and strawberry flavors had a significant impact on the overall cytotoxicity of e-cigarette products.⁶ Another study tested a large variety of e-liquids and discovered that cytotoxic chemicals such as vanillin, ethyl maltol, ethyl vanillin, and menthol were the most frequently found flavor chemicals.⁷ While the cytotoxicity of certain flavor chemicals has been established, little is known about the relationship between e-liquid flavors and health symptoms.

A new generation of e-cigarettes has emerged known as “pod mod devices,” which are popular among youth due to their discrete, sleek designs and easily-operational, high-tech features.⁸ JUUL, a USB drive-shaped pod mod device that debuted in mid-2015, has become dominant in the e-cigarette market, currently controlling a considerable amount of the United States e-cigarette market share and expanding worldwide.⁹ Although JUULs are currently being marketed as an alternative to smoking, studies have shown that underage users ranging from 13–21 years old use JUULs frequently.¹⁰ JUUL revolutionized the industry by using cartridges pre-filled with e-liquid containing nicotine salts, flavorings, and other chemicals.¹¹ JUUL pods contain nicotine salts instead of free-base nicotine used in most e-cigarettes.¹² Nicotine salts allow JUUL to pack their pods with 59 mg/mL of nicotine, which is high for e-liquid standards.¹³ JUUL Pods come in 8 different flavors: Mango, Mint, Menthol, Fruit, Crème, Classic Tobacco, Virginia Tobacco, and Cucumber.¹³ We chose to analyze JUUL flavors due to the limited flavor selection and JUUL’s dominance of the e-cigarette market.

In this study, we proposed the use of social media to deduce possible associations between different JUUL flavors and health issues, as well as track JUUL-related flavor and health trends over time. JUULs and the experiences surrounding them are heavily discussed on social media,¹⁴ which is why we used social media data for our analysis. Social media is an important source of information that can be used to monitor behaviors within a community and ascertain views of the general population in real time.¹⁵ Studies have also shown that web data can more quickly and comprehensively reveal the prevalence of health outcomes compared to surveys.¹⁶ We collected data from Reddit, which is one of the biggest social media platforms. Reddit operates like a forum which is divided into communities called “subreddits,” each covering a specific topic.¹⁷ Reddit users can post and comment on others’ posts, which promotes discussion and dissemination of information.¹⁷ Reddit generally contains less product advertisements and promotions, and more self-reported discussions, making it a good source for reflecting the opinions and experiences of users. Several studies have used Reddit datasets to conduct public health research related to topics like the Ebola outbreak and mental health.^18,19

In addition, there have been a growing body of studies using Reddit and other social media such as Twitter and Instagram to analyze e-cigarette marketing, user opinions, and flavors of e-liquid.^15,20–24 A few studies have used Reddit to analyze JUUL-specific posts in order to understand user perceptions, common user demographics, and reasons for use.^10,25,26 However, most of these studies have not focused on health issues related to e-cigarettes or JUULs. Li et al. is the only study that uses Reddit to qualitatively analyze the potential associations between health symptoms and e-liquid components (such as PG/VG ratios, flavors, and nicotine levels).²⁰ In this work, we extend previous studies by conducting quantitative statistical analysis to determine possible associations between JUUL flavors and certain health problems, using post data from Reddit. Considering the potential changes in the prevalence of different JUUL flavors over time, we conducted temporal analysis to visualize longitudinal change in flavor and health trends related to JUUL. This study’s findings could provide insights about the potential health effects of JUUL flavors, which could accelerate further medical research.

Method

Data collection from Reddit

The data was obtained from the pushshift.io Reddit comments directory, which contains Reddit posts and comments from December 2005 to April 2019.²⁷ Data from January 1^st, 2013 to April 30^th, 2019 was downloaded and processed, and e-cigarette-related posts were obtained by filtering posts with the following e-cigarette-related keywords: “e-cig”, “e-cigs”, “ecig”, “ecigs”, “electroniccigarette”, “ecigarette”, “ecigarettes”, “vape”, “vapers”, “vaping”, “vapes”, “e-liquid”, “ejuice”, “eliquid”, “e-juice”, “vapercon”, “vapeon”, “vapefam”, “vapenation”, and “juul”. We obtained the JUUL related posts by further filtering with specific JUUL-related keywords: “juul”, “juuling”, “juuled”, “juuls”, and “juuler”. Posts that contained irrelevant mentions (i.e. posts mentioning a game designer with the last name “Juul”) were removed by filtering out posts that contained targeted keywords. Emojis, URLs, and non-English posts were removed during the data cleaning. The final JUUL-related dataset contained 106,183 posts, spanning from October 6^th, 2013 to April 30^th, 2019.

Flavors and Health Categorization

The JUUL flavors were used as keywords to obtain the subset of flavor-related JUUL data. The flavor subset contained 5,746 posts and 3,843 Redditors. Similarly, we obtained the health-related dataset using 134 keywords related to health problems and symptoms, which were selected based on user-reported symptoms of e-cigarette use compiled before.²⁸ The keywords were then grouped into 9 categories that were similar to the 12 categories created in a previous study.²⁸ The categories are Throat, Respiratory, Psychological, Neurological, Mouth, Digestive, Cardiovascular, Cancer, and Other (supplemental Table 1). The Mouth & Throat category²⁸ was separated to better reflect different symptoms. Certain Neurological keywords were split into a new Psychological category, and a new Cancer category was added to gauge Reddit user perceptions of cancer in relation to JUUL. Certain health keywords that linguistically appeared within other health keywords, such as “heart” and “heart attack”, were counted separately. All tenses of each keyword were included to avoid misclassification (supplemental Table 1). The final health subset contained 7,927 posts and 5,734 Redditors. All posts were tokenized using Tweet Tokenizer in the Natural Language Processing Toolkit²⁹ before keyword filtering to obtain the best accuracy.

Temporal Analysis

Monthly JUUL-related posts from September 2016 to April 2019 were counted to analyze the overall trend of discussions about JUUL. Data before September 2016 was not included because there were less than 5 posts per month discussing JUUL flavors and health problems. Similar trend plots were created for monthly JUUL post counts for each flavor and health category. Proportion trend plots were created for the flavors and health categories by dividing each monthly count by the total JUUL post count for that month.

Statistical Analysis

To explore potential associations between JUUL flavors and health symptom categories being mentioned by the same users, we first extracted single-flavor users, which we defined as users that only mentioned one flavor among all of their posts, to preserve independence between flavors. We were able to extract a set of 609 single-flavor users. Then, we examined all posts to determine whether the users in this set also mentioned keywords from the health categories. The tobacco flavors, Classic Tobacco and Virginia Tobacco, were combined into a “Tobacco” category for this analysis.

Generalized estimating equation (GEE) models with logit link functions and compound symmetry variance-covariance matrices were used to analyze the potential associations between JUUL flavors and health categories being mentioned by the same users. The GEE model is commonly used to analyze correlated outcomes.^30,31 If a user mentioned multiple health categories, each flavor and health category pair was regarded as a unique observation. Thus, observations from the same user are likely to be correlated. Compound symmetry variance-covariance structure assumes observations from the same user have the same correlation. The logit link function converts the values from the model to the scale of a probability,³² which is how we obtained the probabilities of the same user mentioning certain health categories along with specific JUUL flavors. The parameters in the logit link function were calculated using quasi-likelihood functions. Tukey’s method was used to adjust for multiplicity in pairwise comparisons between flavors. The probability of certain health categories being associated with each flavor in the Reddit data was calculated from estimated parameters in the GEE model. The estimated probability denotes the chance that a flavor and health symptom category are mentioned by the same user. Pairwise comparisons between flavors were used to find significant differences in the probabilities of association. A heatmap was created to better visualize the probabilities of association between JUUL flavors and health categories. The GEE models were conducted using the proc genmod procedure in SAS V.9.4 (SAS Institute Inc., Cary, NC). All tests were two-sided with a significance level of 5%.

Results

Temporal Analysis of JUUL Pod Flavors

The number of JUUL-related posts on Reddit were counted for each month from April 2016 to April 2019, illustrating an exponential increase in JUUL-related posts over time (supplemental Figure 1). There was an increase in the slope of the curve around June 2017, which continued until December of 2018. Since December 2018, the number of JUUL posts has remained steady slightly above 8000 posts per month (supplemental Figure 1).

The most mentioned JUUL flavor on Reddit overall was Mango, followed by Mint. The least mentioned flavors were Virginia Tobacco and Classic Tobacco (supplemental Table 2). This is consistent with the results from our temporal analysis (Figure 1). In October 2018, there was a large spike (orange arrow) in the trends of several JUUL pod flavors.

Figure 1:

(a) Number of monthly posts for each JUUL flavor pod on Reddit over time. (b) Proportion of monthly posts for each JUUL flavor pod on Reddit over time.

Although the posts for each JUUL flavor were increasing over time, the proportion of posts for each flavor after normalizing based on total JUUL posts remained relatively constant with some fluctuations (Figure 1b). The red arrow in Figure 1b indicates a sudden growth in the proportion of posts mentioning the Mango pod beginning November 2016 and growing to a peak just after March 2017. The purple arrow in Figure 1b, indicates a peak in the trends of the Cucumber and Menthol flavors. Overall, the rank of the JUUL flavors discussed on Reddit has remained constant.

Temporal Analysis of Health Categories

Considering the possible association of e-cigarettes with health conditions, we examined health symptoms frequently mentioned together with e-cigarettes on social media. There were nine health categories based on symptoms associated with JUUL on Reddit (Supplemental Table 1). Each health category had several keywords that were the main contributors to the overall trend of that health category (Table 1). The Respiratory category had the greatest number of posts (n = 2,535) and its main keywords were “lung” (n = 1,766), “cough” (n = 630), and “popcorn lung” (n = 154). Other prominent categories were the Throat category that included “throat hit” (n = 1,557) and “throat” (n = 770), the Psychological category (n = 884) that included “anxiety” (n = 405), “stress” (n = 288), and “depression” (n = 197), and the Cardiovascular category (n = 561) that included “heart” (n = 254) and “blood pressure” (n = 66) (Table 1).

Table 1:

Top keywords in each health category

Health Category	Symptom Keyword	# of Posts	Example Post
Respiratory	Lung	1766	“During the weekend I pretty much have it next to me all day and drain about a pod a day…I have a wicked headache and my lungs are aching.”
	Cough	630	“From the JUUL? None, or at least none that I can distinguish from the damage I did smoking. My smokers cough is almost non-existent, and my lung capacity seems to be getting better.”
	Popcorn Lung	154	“She basically told us that JUULs are full of a shitton of nicotine and also the flavoring can possibly give you [popcorn lung] a delightful disease named after the popcorn factory workers who first got it from inhaling the flavoring put on microwave popcorn.”
Throat	Throat Hit	1557	“It’s far from a vaping experience, it is much more similar to a smoking experience though, strong throat hit and very strong nic hit.”
	Throat	770	“It definitely outperforms any cigalike i’ve encountered, but it is quite harsh. Even holding for 3+ seconds on a mouth to lung hit left my throat burning on inhale.”
Neurological	Headache	137	“…I’ve never experienced this myself but it usually kills it for me from the headache I get from chain vaping the JUUL.”
	Sensitive	113	“Sometimes people have sensitivity to higher pg levels…the ejuice made me want to vomit. I believe that it was due to the high pg level…I also used to vape on one of those JUUL sticks.”
	Dizzy	89	“The JUUL already has me hypersalivating feeling dizzy unless I control myself and put it down.”
Psychological	Anxiety	405	“Nicotine, especially the JUUL, can at times amplify my anxiety so I would not recommend using the JUUL…”
	Stress	288	“…The juul has helped me with stress…”
	Depression	197	“I have anxiety depression and I wish I never started smoking cigs. The JUUL helped me cut down a lot on cigs and helped eliminate most of the bleh feeling they give…Having a little escape is nice and the JUUL is a much better vice than many on the market, especially if your psych leans on the benzo side for relaxation.”
Cardiovascular	Heart	254	“Things like JUUL literally contain ingredients like nicotine, which have been proven to increase the risk of heart disease and possibly [lung] cancer.”
	Blood Pressure	66	“Yeah after hitting my JUUL alot I figured out the reason what was causing my breathing issues was the 50 nic was raising my blood pressure and making it harder for me to breathe…”
Digestive	Stomach	158	“Have you ever smoked a juul? Cigarettes don’t give me stomach aches but a JUUL wrecks me every time.”
	Vomit	156	“The JUUL mint has been making me nauseous to the point of actually vomiting lately. Other flavors are fine though.”
Mouth	Teeth	183	“I switched to the JUUL a year or so ago…my teeth are very white and have very nice gums according to my dentist 1 month ago, I can’t remember the last time I heard that, if ever.
	Lips	108	“JUUL’s just not a sustainable pod/device ecosystem for myself. high nic + spit back = numb lips/sucks.”
	Tongue	99	“Not a sore throat, sore tongue. I never had this problem before until I started using a JUUL when I decided to stop smoking cigarettes three months ago.”
Cancer	Cancer	642	“You could make an argument that the JUUL is safer as you need less vapor to fullfil your nic craving…if your worried about cancer vaping is way down on the list of things that can contribute to you getting cancer.”
	Lung Cancer	172	“I’ve gotten at least 12 chain texts from people with mutual friends that now have lung cancer from only juuling....I am definitely freaking out and have thrown my juul away cause I can’t shake it. However, it seems odd that this happened to these individuals after only a few months of juuling…”
Other	Hurt	499	“…You can’t taste anything, it has that spicy nic taste that makes my chest hurt.”
	Skin	317	“I impulsively purchased a new JUUL, almost immediately I noticed my skin felt tight and irritated. Overnight new small, flat red spots appeared--they hurt and feel as if they have their own pulse at times though they were not yet pimples.”
	Nausea	184	“The classic menthol has a strange aftertaste…after about 3–4 pulls I started getting a headache and started getting nauseous. I swapped in a mint pod and felt fine. Couple of hours later I tried a couple of pulls from the classic menthol pod and started getting the same headache and nausea.”

The number of posts mentioning each health category on Reddit has been increasing over time (Figure 2a). The Respiratory and Throat health categories were mentioned the most frequently over time, while the Digestive category was mentioned the least. After adjusting for the overall JUUL-related post count, the proportion of posts related to the Throat category was decreasing over time (Figure 2b). The proportion of posts related to the Respiratory category was slightly increasing starting from late 2018. The red arrow in Figure 2a and 2b indicate the same peak in the Respiratory and Cancer health categories in early 2018.

Figure 2:

(a) Number of monthly posts for each health category on Reddit over time. (b) Proportion of monthly posts for each health category on Reddit over time.

Potential Associations between JUUL Pod Flavors and Health Categories

The probability of each JUUL flavor being mentioned with different health categories by the same user was calculated in order to examine which flavors might be associated with health symptom categories on Reddit (Figure 3). The Throat category had relatively high probabilities of association with all the JUUL flavors. The Menthol flavor had the highest probability of association with the Throat category (0.326), followed by Mint (0.311) and Mango (0.299) flavors. The Fruit flavor had a relatively higher probability of association with the Respiratory category than other flavors. In the Psychological category, the Cucumber flavor had a relatively high probability of association (0.130) compared to other JUUL flavors. In the Digestive category, the Crème flavor had a higher probability of association (0.107) compared to other flavors. In the Cardiovascular category, the combined tobacco flavors had a higher probability of association (0.161) compared to other flavors.

Figure 3:

Heatmap illustrating the probabilities of association between JUUL pod flavors and health categories.

Figure 4 shows the results from pairwise comparisons between each flavor’s probability of association with health categories. Five out of the nine health categories had statistically significant associations with e-liquid flavors. Mango, Mint, and Fruit flavors had significantly higher probabilities of association with the Respiratory category than the Menthol flavor (Figure 4a). The Crème flavor had a higher probability of association with the Digestive category compared to Mango and Menthol flavors (Figure 4b). Compared to the Mango flavor, the Cucumber flavor had a higher probability of association with the Psychological category (Figure 4c). The tobacco flavors had a higher probability of association with the Cardiovascular Category than Menthol, Mint, and Mango flavors (Figure 4d). Reddit users were more likely to post about the tobacco, Fruit, and Mango flavors with cancer-related topics (Figure 4e). All JUUL flavors had relatively high but similar probabilities of association with the Throat category (Figure 4f).

Figure 4:

Bar charts illustrating the probabilities of association between the JUUL pod flavors and health categories, including the (a) Respiratory, (b) Digestive, (c) Psychological, (d) Cardiovascular, (e) Cancer, and (f) Throat health categories. There were no significant pairwise comparisons between flavors for the Throat, Mouth, Neurological, and Other health categories. Throat was included because the probabilities of association were relatively high for all the flavors.

Error bars represent 95% CI. ** indicates p<0.01, * indicates p<0.05

Discussion

This study provides valuable insights about the potential associations between specific JUUL flavors and categories of health problems. Through temporal analysis, we found that the Mango flavor was consistently the most mentioned JUUL flavor over time, and the Respiratory category was mentioned the most among all examined health categories over time. The association analysis between JUUL flavors and health categories indicated that some JUUL flavors have a greater potential association with certain health problems than others; for example, the JUUL Fruit, Mango, and Mint flavors were more likely to be associated with the Respiratory category than the Menthol flavor. This study demonstrates that discussion forums like Reddit could be a good resource for supporting previous research findings and catalyzing new research that is based on user experiences and opinions.

JUUL was a good candidate for this analysis because the pods contain the same Propylene Glycol (PG) to Vegetable Glycerin (VG) ratio.¹³ The nicotine level of JUUL has remained constant at 5%, until the company released new 3% pods in August 2018.¹¹ Since the majority of our data is from before August 2018, we assumed that the nicotine level and PG/VG ratio were consistent across all JUUL flavors. This strengthens our results, as differences in probabilities of association for certain flavors could be attributed more to the different flavor chemicals.

We found that the Respiratory category had greater probabilities of association with the Fruit, Mango, and Mint flavors. This is consistent with a study showing higher concentrations of cytotoxic flavor chemicals (vanillin, ethyl maltol, benzyl alcohol) in the Mango and Mint flavors, which could indicate their possible association with Respiratory symptoms.³³ The Fruit flavor was found to contain menthol and benzaldehyde PG acetals, which are cytotoxic in aerosol and could also lead to respiratory symptoms.³³ Upon further qualitative analysis of posts in the Respiratory category, we noticed that posters mentioned that they were coughing or had aching lungs because of the JUUL (Table 1). However, some posters also described that their lungs felt better when they used the JUUL, in comparison to smoking. Therefore, this suggests that symptom descriptions making up this potential association between Fruit, Mango, or Mint users and Respiratory symptoms could be negative, or positive in comparison to smoking.

The combined tobacco flavors had the highest probability of association with the Cardiovascular category. Both Virginia Tobacco and Classic Tobacco have low concentrations of flavor chemicals, with Classic Tobacco containing low levels of benzyl alcohol and Virginia Tobacco containing negligible levels of any flavor chemicals.³³ Therefore, cardiovascular symptoms might be due to high nicotine levels in the tobacco-flavored JUUL pods. In addition, most users of the tobacco flavors switched from smoking or are dual users, because the tobacco flavors have a similar taste to combustible cigarettes.³⁴ These users could have developed these symptoms from traditional cigarette smoking before switching to JUUL.

The Crème flavor had the highest probability of association with the Digestive category. The Crème flavor contains high levels of vanillin, which is cytotoxic in aerosol. ^33,35 However, there is not much research on how this could affect digestive symptoms. Some users have described trying the Crème flavor and then vomiting, which could be how this association appeared. The Cucumber flavor had the highest probability of association with the Psychological category. Sentiments about psychological symptoms varied. Many posts wrote about anxiety relief after using the JUUL, but a large portion stated that JUUL increased their anxiety. However, research has shown that although nicotine users believe smoking eases their anxiety, this is likely due to their nicotine addiction.³⁶ This may suggest that Cucumber’s possible association relates to its addiction potential. The Cucumber flavoring is made up of mostly menthol, which has been shown to increase nicotine uptake in the blood.³⁷

The Cancer category was created to see user perceptions about cancer on Reddit and what flavors users mention with the hypothetical notion that e-cigarettes could increase the risk of cancer. Cancer develops over time, making it difficult to track on social media, and many of the claims could be inaccurate. Nevertheless, Fruit, Tobacco and Mango had high probabilities of association with the Cancer category.

Based on our temporal analysis of JUUL flavors discussed on Reddit, Mango was consistently the most mentioned flavor over time, followed by Mint. This is consistent with a previous JUUL-related study on Reddit, which counted the total mentions for each flavor.²⁵ A survey study also showed that among 15–17 year-olds, Mango and Mint were used more often in the past 30 days.³⁸ The tobacco flavors had few mentions in our study, which is consistent with the survey study.³⁸ This illustrates that Reddit is a good indicator of flavor use frequency by youth and young adults. The similarity to the youth survey study could indicate that the population of Reddit users discussing JUUL tend to be younger. This is supported by a study which showed that the age of Redditors participating in JUUL subreddit discussions ranged from 13 to greater than 21 years old.¹⁰

The popularity of the Mango and Mint flavors also parallels the flavors of e-juices that are generally the most popular. According to a study about e-cigarette flavors, the most popular flavors are in the Fruit category, followed by the Crème category and Menthol category. ³⁹ The eight flavors in the present study seem to be broadly representative of the e-cigarette flavor categories, as Mango and Fruit represent the Fruit category, Crème represents the Crème category, Mint and Menthol represent the Menthol category, Classic Tobacco and Virginia Tobacco represent the Tobacco category, and Cucumber potentially represents the Sweet category. Therefore, our results seem to confirm more general e-cigarette research.

In our temporal analyses of JUUL flavors and health categories, there were no large shifts in the rank of flavors or the perception of health symptoms, but there were certain events that caused fluctuation in the data. The Mango flavor had a sudden growth in mentions (red arrow in Figure 1b), which was most likely due to the first release of Mango pods to stores.⁴⁰ The Cucumber and Menthol flavors also had a peak (purple arrow in Figure 1b) after they were released in October 2017.⁴¹ Several major peaks corresponded with news events related to JUUL that were heavily discussed. The spike in Figure 2 (orange arrow) was due to discussion regarding JUUL suspending its retail sales of Mango, Crème, Cucumber, and Fruit to prevent underaged teenagers from obtaining flavor pods.⁴² The peak in Figure 3 (red arrow) came from a tweet that was spread by young adults about several peers being diagnosed with lung cancer. The frenzy originated with a viral tweet that contained screenshots of group chat messages talking about friends being diagnosed with cancer. This tweet was eventually debunked by medical professionals and JUUL Labs, which explains the decrease in mentions shortly after.⁴³ These instances show that Reddit is a good indicator of events over time and more importantly, could be used in a predictive capacity to forecast future rises and falls in certain topics.

There were several limitations in this study. Firstly, solid correlation or causal relationships between the JUUL flavors and health symptoms cannot be established because the symptoms and JUUL flavors are just mentioned by users on social media; therefore, the posters might not necessarily have these symptoms or use these JUUL flavors. However, social media data can provide timely and reasonable estimates of health prevalence and potential associations of e-cigarette flavors with health symptoms. In our Reddit dataset, many users were seeking advice and sharing their experiences, which helped us explore potential associations. We also cannot verify that single-flavor users as we defined them are in fact only using one flavor. Our current analysis only considers the potential association of a single JUUL flavor with health categories mentioned by the same user. We did not distinguish posts expressing JUUL flavors causing or easing certain symptoms from posts expressing irrelativeness of these flavors and symptoms, because we did not conduct syntactical or sentiment analysis for posts. This might cause potential bias in our data analysis but will be considered in future studies. In addition, e-juice flavoring is only one factor that could affect the popularity of a certain e-cigarette product and its association with health categories. ⁴⁴ Therefore, our research could be generalizable to similar pod-mod devices but might not be generalizable to other e-cigarettes with different mechanisms, like box mods or tank devices. Finally, we could not take into account certain demographic information such as age, gender, ethnicity, how long the user has smoked combustible cigarettes, how often the user vapes, etc., because Reddit does not provide this information to the public. In the future, sentiment analysis could be performed to better assess user opinions about JUUL or e-cigarettes with different flavorings. It would also be worthwhile to examine users that switch flavors frequently. On January 2nd, 2020, the U.S. Food and Drug Administration (FDA) issued a policy to prohibit the sale of unauthorized flavored cartridge-based e-cigarette products that are appealing to youth, except tobacco and menthol flavors.⁴⁵ Thus, the prevalence of e-cigarette flavors might change with the enforcement of the FDA policy, which will be worth exploring in future studies.

Despite the limitations, this study is the first to quantitatively measure potential associations between JUUL flavors and health symptom categories using JUUL-related Reddit posts. Even with the flavor ban, JUUL is still a popular e-cigarette brand in the United States and is currently expanding its market worldwide with similar flavors. Health symptoms related to JUULing should be further explored, and the results from this study could serve as a springboard for related research about different e-cigarettes, such as disposable e-cigarettes that have similar flavors to JUUL.⁴⁶ E-cigarettes and their flavorings are associated with certain health issues, as illustrated in this study, and it is necessary to analyze them to provide insights about how to regulate these e-cigarettes and flavors in the future.

What this paper adds.

• By conducting temporal analysis with normalization, we show that the discussion about the eight JUUL flavors and nine health symptom categories on Reddit remains relatively constant over time.

• Based on user-generated Reddit posts, our results show that different JUUL flavors have different probabilities of association with health symptom categories.

• This study illustrates how to conduct quantitative and qualitative analyses of online resources, such as social media, that provide insights into community perspectives and behaviors in real time.