Quality and Popularity Trends of Weight Loss Procedure Videos on TikTok

Abstract

Introduction

The use of social media as a medical information tool parallels rising obesity rates. TikTok, the popular video-sharing platform, contains nearly 99,000 videos hashtagged “weightloss.” Prior studies have analyzed the quality of medical information on TikTok in other areas of medicine. However, the quality of videos regarding weight loss procedures has not yet been determined.

Methods

Hashtags encompassing three weight loss modalities were searched using TikTok’s algorithm. The first 50 videos meeting inclusion criteria for each modality were considered. Two independent reviewers categorized videos and assessed their content quality using DISCERN. Quality scores and popularity were compared between videos sources, modalities, and content categories.

Results

Of 150 videos included, 20.7% were created by physicians versus 79.3% by non-physicians (p < 0.001). The average DISCERN score for physician-created content was significantly higher than that of non-physicians (p < 0.001), despite significantly less popularity (p < 0.002). The 50 most popular videos had significantly lower DISCERN scores than the 50 least popular (p < 0.02). The average DISCERN score for endoscopic sleeve gastroplasty (ESG) videos were significantly higher than videos related to Roux-en-Y gastric bypass (RYGB) and vertical sleeve gastrectomy (VSG) (p < 0.001). VSG-related videos were significantly more popular than RYGB- and ESG-related videos (p < 0.001 and p < 0.001). Finally, educational videos had significantly higher DISCERN scores than weight loss transformation and personal experience videos (p < 0.001).

Conclusion

Videos on TikTok related to weight loss procedures are poor, and greater popularity trends with lower quality. Assessment of content can encourage viewers to seek better information and allow providers to improve patient information tools.

Graphical Abstract

Introduction

TikTok has developed into one of the most popular social media platforms across the world since its launch in 2016, currently boasting over one billion monthly active users [1]. Over half of these users are between the ages of 10 and 29 years old, though engagement with those over 50 continues to grow [2]. The application allows users to publish short-form videos, which, although initially produced for entertainment purposes, are now being used for education on a wide variety of topics including health. Internal data gathered by Google suggests that nearly 40% of canvassed “Gen Z” members (those born between 1997 and 2012) prefer TikTok as a search engine over Google [3]. Consistent with this trend, TikTok overtopped Google as the most visited website of 2021 [4]. Accordingly, the evolution of social media as a search engine has piqued the interest of medical professionals and laypeople alike, who are increasingly utilizing TikTok to share information related to health and medicine [5].

Health information is some of the most popular content on TikTok. Within 6 months of the pandemic onset, the hashtags “medicine” and “doctor” yielded 1.4 billion and 6.7 billion views, respectively [5]. The popularity of this content may be attributed to the platform’s emergence during the COVID-19 pandemic, though medical content has expanded beyond the now ubiquitous viral infection [6]. For example, videos discussing approaches to and health implications of excess body weight are numerous. Unsurprisingly, videos characterized as “fitness/sports” and “recipes/cooking” comprise two of the top ten most-viewed content categories on TikTok [7]. Across the platform, the hashtag “weightloss” has been used in nearly 99,000 videos collectively viewed over 58 billion times [8]. A small but significant portion of these “weightloss”-tagged videos—approximately 3000—reference one of the following weight loss surgeries or procedures: vertical sleeve gastrectomy (VSG), Roux-en-Y gastric bypass surgery (RYGB), or endoscopic sleeve gastroplasty (ESG). Despite the popularity of these interventions in both the clinical setting and on social media, the quality of these videos on TikTok has yet to be analyzed.

Methods

Hashtags encompassing the three weight loss modalities of interest—“gastricsleeve,” “gastricsleevesurgery,” “vsg,” “vsgcommunity,” “rny,” “rnycommunity,” “gastricbypasssurgery,” “gastricbypass,” “endoscopicsleeve,” and “endoscopicsleevegastroplasty”—were entered into TikTok’s search algorithm and yielded over 3000 videos. The first 50 algorithm-generated, resultant videos meeting inclusion criteria for each weight loss modality were considered. Duplicated videos, non-English language videos, and videos without audio or visual explanation were excluded during the screening phase to yield an equal amount of videos to be assessed in each category (Fig. 1). For each video, descriptive statistics were recorded. These included content creator (physician vs. non-physician), views, likes, shares, uploader’s gender, video upload date, and video length. Popularity was recorded as views per day. We then modified Naseer et al.’s method of visual categorization and stratified videos into groups 1, 2, and 3[9]. Group 1 videos had minimal supplementary visuals, while group 2 had moderate supplementary visuals and group 3 was rich in supplementary visuals. Two independent reviewers, A. L. and A. H., analyzed video content and descriptions to categorize and score each video using DISCERN, a validated tool assessing the quality of consumer health information. The DISCERN questionnaire consists of 16 questions on a continuous rating scale of 1 to 5. A higher cumulative DISCERN score correlates with higher quality and reliability of the assessed source. The grade of each video was averaged between the two reviewers. Disputes regarding video categorization were resolved by a third independent reviewer, R. Z. S. Quality scores and popularity were compared between videos produced by physician and non-physician creators. These metrics were also compared between modalities and content categories. Cohen’s kappa coefficient was used to assess the inter-rater reliability. The one-way analysis of variance (ANOVA), the two-sample T test, assuming both equal and unequal variances where appropriate, and Pearson’s chi-squared test were used to determine statistically significant groups.

Fig. 1.

Study identification and selection overview. Videos were viewed based on TikTok’s hashtag search result algorithm

Results

Included Video Characteristics

A total of 150 videos meeting the inclusion criteria were included in our final analysis. The videos had a total of 105,599,969 views, amassing 11,686,009 likes and 113,314 shares (Fig. 2). Quality scores utilizing DISCERN ranged from 22 to 58, with a mean DISCERN score of 31.57 with high inter-rater reliability (Cohen’s kappa > 0.9).

Fig. 2.

Hashtag characteristics. Overview of overall hashtag characteristics and included video descriptive analysis

Content Creator Characteristics

When assessing content creator characteristics, more videos were published by females (n = 124, 82.7%) compared to their male counterparts (n = 36, 17.3%). Content creators were more commonly non-physicians (n = 119, 79.3%) rather than physicians (n = 31, 21.7%). Among these two groups, non-physicians’ videos were significantly more popular (p < 0.01) and had significantly more shares (p < 0.05) (Fig. 3). However, videos produced by physicians were of significantly longer duration (p < 0.001) and had significantly higher DISCERN scores (p < 0.001). After searching for videos via weight loss modality hashtags, we found that resultant videos primarily centered around weight loss transformation content (n = 67, 44.7%), followed by educational content (n = 45, 30.0%), and then personal experience anecdotes (n = 38, 25.3%) (p < 0.001). There was no significant difference in the number of likes, shares, or popularity across videos of different content types. However, DISCERN scores were significantly higher for videos revolving around educational content compared to other video types (p < 0.001). Videos categorized as educational were also significantly longer in duration compared to those of weight loss transformations and personal experiences (p < 0.001).

Fig. 3.

Popularity and quality of content on TikTok. Popularity was assessed and categorized by surgery modality, content creator profession, and content type

DISCERN Question Performance

Physician creators scored significantly higher as compared to non-physician creators on DISCERN questions 1 (“Are the aims clear?”) (p < 0.001), 2 (“Does it achieve its aims?”) (p < 0.001), 3 (“Is it relevant?”) (p < 0.001), 4 (“Is it clear what sources of information were used?”) (p < 0.001), 6 (“Is it balanced and unbiased”) (p < 0.001), 7 (“Does it provide details of additional sources?”) (p < 0.001), 8 (“Does it refer to areas of uncertainty?”) (p < 0.01), 9 (“Does it describe how each treatment works?”) (p < 0.001), 10 (Does it describe the benefits of each treatment?”) (p < 0.001), 11 (“Does it describes the risks of each treatment?”) (p < 0.001), 12 (“Does it describe what would happen if no treatment is used?”) (p < 0.001), 14 (“Is it clear that there may be more than one possible treatment choice?”) (p < 0.001), and 15 (“Does it describe what would happen if no treatment is used?”) (p < 0.001) (Table 1). Furthermore, visual category scores were significantly higher for videos produced by non-physician providers compared to videos produced by physicians (p < 0.05). In addition, DISCERN scores were significantly higher for videos that were greater than 30 s in duration (p < 0.001).

Table 1.

DISCERN question performance analysis. Comparing the DISCERN question performance between physicians and non-physicians

DISCERN questions	Combined average	Non-physician	Physician
1) Are the aims clear?	3.320	3.172	3.887***
2) Does it achieve its aims?	3.350	3.235	3.790***
3) Is it relevant?	3.253	3.155	3.629***
4) Is it clear what sources of information were used to compile the publication?	1.267	1.076	2.000***
5) Is it clear when the information used or reported in the publication was produced?	1.907	1.932	1.806
6) Is it balanced and unbiased?	1.410	1.205	2.193***
7) Does it provide details of additional sources of support and information?	1.277	1.067	2.080***
8) Does it refer to areas of uncertainty?	1.533	1.415	1.983**
9) Does it describe how each treatment works?	2.180	1.840	3.483***
10) Does it describe the benefits of each treatment?	2.737	2.563	3.403***
11) Does it describe the risks of each treatment?	1.537	1.331	2.322***
12) Does it describe what would happen if no treatment is used?	1.900	1.727	2.565***
13) Does it describe how the treatment choices affect overall quality of life?	2.913	2.916	2.903
14) Is it clear that there may be more than one possible treatment choice?	1.56	1.210	2.903***
15) Does it provide support for shared decision-making?	1.423	1.147	2.484***

p-value < 0.05*, p-value < 0.01**, p-value < 0.001***

Surgery Modality Popularity and Content Quality

Of the 150 videos equally distributed across VSG, RYGB, and ESG, videos with VSG-related hashtags had significantly more likes (p < 0.05) and shares (p < 0.001) and were more popular (p < 0.001) than both RYGB and ESG videos (Fig. 3). Notably, the least popular of the surgery modality videos, ESG, had a significantly higher mean DISCERN score than both alternative surgery videos (p < 0.001). Furthermore, analysis of all 150 videos, irrespective of surgery type hashtag, demonstrated that the top 50 most popular videos had significantly lower DISCERN scores than the least 50 popular videos (p < 0.05) (Fig. 4).

Fig. 4.

Discern score vs. popularity. Comparison of DISCERN score relative to popularity of video stratified into three groups— top 50, middle 50, and least 50 popular videos

Discussion

Video-sharing platforms have changed the way in which healthcare information is communicated, shared, and seen [10]. With rising rates of obesity and associated psychiatric comorbidities, platforms like TikTok allow patients to search for information about their medical conditions and therapeutic procedures in an unrestricted way. Previous studies have analyzed the impact of social media on mental health in those with weight issues and demonstrated a need for providers to be aware of their patients’ social media use [11]. Given TikTok’s rapid rise in popularity, medical practitioners have appraised the quality of information in the fields of ophthalmology and plastic surgery. However, to date, no studies have focused on information related to weight loss procedures such as bariatric surgeries or endoscopic procedures [6, 9].

Social media is often consulted as an alternative resource for those who want simple, first-person accounts of medical conditions or procedures, and can result in patients forgoing traditional medical consultation [12]. Consistent with this trend, our study found that 79.3% of the videos included in our analysis were created by non-physicians. In addition to greater popularity, videos from non-physicians scored significantly higher in visual quality and were shorter in duration. This higher visual quality, irrespective of content quality, may be contributing to the popularity and dissemination of videos from non-physician creators. In fact, higher content quality does not increase video popularity at all; videos with the highest DISCERN scores were significantly less popular. On average, the top 50 most popular videos across all weight loss modalities and content types had the lowest DISCERN scores, whereas the least 50 popular videos had the highest DISCERN scores. Regarding content categorization, educational videos were primarily created by physicians and, by extension, had the highest average DISCERN scores. Collectively, our results suggest that viewers are not engaging with videos based on content quality or categorization, but rather, with videos that have more visuals and entertainment value.

While there were multiple DISCERN questions on which physicians outperformed non-physicians, a subset of these questions warrant additional consideration. Questions of note include numbers 4 (“Is it clear what sources of information were used?”), 6 (“Is it balanced and unbiased”), 11 (“Does it describe the risks of each treatment?”), and 15 (“Does it describe what would happen if no treatment is used?”). These questions highlight enhanced risk of medical misinformation and the need for physicians to recognize the information easily available to patients. Potentially misleading or false information can lead to inappropriate or unrealistic expectations regarding treatment outcomes and lower patient satisfaction [13].

When assessing the different procedural weight loss modalities, a similar pattern was observed between video popularity and content quality. VSG-related videos, which were the most popular and shared, were significantly lower in quality compared to the other two surgical weight loss modalities analyzed. This popularity is likely, in part, due to the frequency at which the procedure is being performed; for example, in 2020, VSG was performed over 122,000 times and comprised over 60% of all weight loss surgeries [14]. Moreover, there were only 15,000 ESG procedures performed that year [14]. Such novelty of ESG explains why most ESG-related content on TikTok had been produced by physicians and primarily focused on providing educational material. The large proportion of physicians creating ESG-related videos could further explain why the average DISCERN score of those videos is significantly higher than the other surgical modalities. These findings suggest that encouraging physicians to create content can remedy the low-quality videos circulating on TikTok.

As with other studies using a comparable methodology, the scope of our study is limited. Of nearly 3000 yielded videos, only the first 50 meeting inclusion criteria were considered for each weight loss modality. This process excluded videos in Chinese, Spanish, and other popular languages on TikTok, which comprise a large portion of the platform’s content. Such a small sample size may not be representative of the broader content on the platform, and it is impossible to know why these videos were listed first. TikTok’s algorithm for resultant video prioritization is not publicly available information. The inability to know or predict changes to this algorithm challenges the generalizability of our results. Finally, this methodology only accounts for videos associated with the hashtags of interest. Our search, though comprehensive, included representative hashtags of the surgeries and procedures of interest but cannot account for alternative hashtags or content that was untagged. The continual addition of videos uploaded to TikTok may change the average quality of information regarding each weight loss modality. Thus, we encourage physicians to continue creating and monitoring for evidence-based videos on TikTok.

Conclusion

Given TikTok’s proliferation, assessment of its content is essential for understanding the application’s impact as a medical information tool. Our results show that videos related to weight loss procedures are poor and that greater popularity trends with lower quality. Though creators produce content at their discretion, critical appraisal of content can encourage viewers to seek more accurate information and allow providers to develop more engaging patient information tools. Indeed, physicians have an obligation to create content that reaches patients where they are—namely, on social media—to enhance availability of accurate medical information on weight loss procedures.

Data Availability

The data that support the findings of this study are available on request from the corresponding author, RZS.

Declarations

Ethics Approval and Consent to Participate

This article does not contain any studies with human participants or animals performed by any of the authors. For this type of study, formal consent is not required.

Conflict of Interest

These authors disclose the following: Reem Z. Sharaiha is a Consultant for Boston Scientific, Cook Medical, Lumendi, and Olympus.