Social media sentiment analysis to monitor the performance of vaccination coverage during the early phase of the national COVID-19 vaccine rollout

Abstract

Background and objective

Social media sentiment analysis based on Twitter data can facilitate real-time monitoring of COVID-19 vaccine-related concerns. Thus, the governments can adopt proactive measures to address misinformation and inappropriate behaviors surrounding the COVID-19 vaccine, threatening the success of the national vaccination campaign. This study aims to identify the correlation between COVID-19 vaccine sentiments expressed on Twitter and COVID-19 vaccination coverage, case increase, and case fatality rate in Indonesia.

Methods

We retrieved COVID-19 vaccine-related tweets collected from Indonesian Twitter users between October 15, 2020, to April 12, 2021, using Drone Emprit Academic (DEA) platform. We collected the daily trend of COVID-19 vaccine coverage and the rate of case increase and case fatality from the Ministry of Health (MoH) official website and the KawalCOVID19 database, respectively. We identified the public sentiments, emotions, word usage, and trend of all filtered tweets 90 days before and after the national vaccination rollout in Indonesia.

Results

Using a total of 555,892 COVID-19 vaccine-related tweets, we observed the negative sentiments outnumbered positive sentiments for 59 days (65.50%), with the predominant emotion of anticipation among 90 days of the beginning of the study period. However, after the vaccination rollout, the positive sentiments outnumbered negative sentiments for 56 days (62.20%) with the growth of trust emotion, which is consistent with the positive appeals of the recent news about COVID-19 vaccine safety and the government's proactive risk communication. In addition, there was a statistically significant trend of vaccination sentiment scores, which strongly correlated with the increase of vaccination coverage (r = 0.71, P<.0001 both first and second doses) and the decreasing of case increase rate (r = -0.70, P<.0001) and case fatality rate (r = -0.74, P<.0001).

Conclusions

Our results highlight the utility of social media sentiment analysis as government communication strategies to build public trust, affecting individual willingness to get vaccinated. This finding will be useful for countries to identify and develop strategies for speed up the vaccination rate by monitoring the dynamic netizens' reactions and expression in social media, especially Twitter, using sentiment analysis.

1. Introduction

In response to the prolonged COVID-19 pandemic, the World Health Organization (WHO) and its partners have to speed up vaccines' rapid development and deployment to protect from the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) [1], [2], [3]. Currently, eight COVID-19 vaccines have been granted emergency use of authorizations by WHO, such as Sinovac, Sinopharm/BIBP, Moderna, AstraZeneca/Oxford, Johnson & Johnson, Covishield, and Pfizer/BioNtech.[4] Various other vaccines are also being administered in many countries under their national regulatory authorities [1].

On January 13, 2021, Indonesia began its COVID-19 vaccination program, with the first phase targeting health workers and frontline workers [5]. The government has set a target of getting 181.5 million people (70% of the total population) to get vaccinated to create herd immunity within 15 months [5]. However, despite having the second-highest number of COVID-19 cases and deaths in Asia, only 10.4 million people (3.7%) were fully vaccinated against COVID-19 [6]. Moreover, with the current pace, herd immunity may not be reached soon in Indonesia, providing opportunities for broader circulation of new variants emerging [1].

The Indonesian government is working to increase the number of daily vaccination shots to reach the target. Besides dealing with the global vaccine suppliers and developing the national vaccine industry to meet the required stocks, WHO's Strategic Advisory Group of Experts (SAGE) has advised all countries to develop a strategy to increase vaccine uptake and address vaccine hesitancy [3]. Moreover, there has been a surge of COVID-19 vaccines misinformation and conspiracy theories surrounding the efficacy and safety of the vaccines that potentially reduce individual willingness to get vaccinated [7]. Therefore, it is critical to monitor the spread of the COVID-19 vaccine's misleading information and mitigate its impact to reinforce public confidence and address vaccine hesitancy successfully.

Social media has been frequently used to develop a large-scale real-time tracking tool on disease outbreaks and the government epidemic control measures during the global pandemic [8,9]. Social media also become a crucial communication platform to disseminate either valid or misleading information faster than traditional news reporting [10,11]. It can also be effectively used to understand public responses, which can potentially help the nations influencing public behavioral response as mitigation strategies to combat the pandemic [12], [13], [14]. Among other social media, Twitter was the leading platform used by people to express their perceptions or behaviors toward health information [9,[15], [16], [17]]. Twitter study is also part of a supply-based "infodemiological" study which offers the opportunity to analyze people's needs through their health information-seeking behavior, including vaccine uptake [18]. A few studies using Twitter have been conducted to assess vaccine uptake on influenza, A(H1N1), and human papillomavirus (HPV) [19], [20], [21]. With the number of Twitter active users in Indonesia that reached 22.8 million users [22], enabling timely monitoring on public reactions to the government risk messages and the immediate actions to clarify misinformation [23].

Moreover, a recent survey of Indonesian Twitter users showed that 77.9% of adults in Indonesia (18-44 years old) used Twitter [24], which is most notable for reaching as the government targeted intervention population for COVID-19 vaccines was the working-age population. We, therefore, used these benefits to analyze the public sentiments on Twitter in response to the COVID-19 vaccine rollout focusing in Indonesia. We hypothesized that public sentiment would correlate with the national vaccination coverage, contributing to the decline of COVID-19 case increase and case fatality rate.

2. Methods

The approach to the methodological framework for data collection and analysis is summarized in Fig. 1 .

Fig. 1.

The Methodological Framework for Data Collection, Preprocessing, and Analytics.

2.1. Data collection

We divided our data collection into two steps. First, we collected Twitter conversations of Indonesian Twitter users using a Twitter monitoring and analytics platform called Drone Emprit Academic (DEA) [25]. DEA is a big data technology based on artificial intelligence, specifically machine learning and natural language processing, utilized for social media monitoring and analytics [26]. The search criteria included all tweets related to COVID-19 filtering with the selected keyword #vaksin (vaccine). All tweets were retrieved over 180 days, from October 15, 2020, to April 12, 2021, to observe the temporal dynamics of public reactions before and after the vaccination rollout in Indonesia [27]. Second, we obtained the data relating to COVID-19 vaccination coverage from the MoH official website [28]. Meanwhile, COVID-19 case increase and case fatality rate were retrieved from the KawalCOVID19 crowdsourced database. The case increase rate was determined as the ratio of the new cases to the total number of cases. The case fatality rate was calculated as the ratio of those who died from COVID-19 to the number of COVID-19 patients [29]. We only obtained 90 days of reporting data from the MoH official website and KawalCOVID19 database starting from January 13 to April 12, 2021, which corresponded with the early phase of the national COVID-19 vaccination rollout.

2.2. Data analysis

We processed all filtered tweets by removing irrelevant attributes, including slang words/sarcasm, short terminology, and stop words. We analyzed the frequency, sentiments, and trends of all processed tweets (including mentions, retweets, replies, and favourites) using DEA Twitter analytics features similar to those used in the previous study [14]. In addition, we also calculated the number of interactions of certain tweets (interaction rate) to understand the public concern or engagement toward COVID-19 vaccine in the selected period. Interaction rate was calculated as the total number of replies, retweets, and favourites a tweet receives divided by the total number of mentions. The interaction rate is high if the total number of mentions is lower than the total number of replies, retweets, and favourites [26]. The detailed interaction rate formula is as follows:

$$interactionrate=\frac{\left(tota{l}_{replies}+tota{l}_{retweets}+tota{l}_{favourites}\right)}{tota{l}_{mentions}}$$

DEA sentiment analysis used Naïve Bayes (Adaptive Multiple Model) methods to classify a word as positive, negative, or neutral with an accuracy of 90.26% [30]. We calculated the vaccination sentiment score as the relative difference between positive sentiment and negative sentiment tweets [21], which is defined as

$$sentimentscore=\frac{\left(n_{positive}-n_{negative}\right)}{\left(n_{positive}+n_{negative}+n_{neutral}\right)}$$

We extracted all hashtags from the tweets corpus database to create the hashtag list. A word cloud (also known as a tag cloud) was formed to visualize the high-frequency words for each sentiment category. We also performed emotion analysis based on Plutchik's Wheel of Emotions (joy, fear, anticipation, anger, disgust, sadness, surprise, and trust) to identify the predominant of tweets [26].

2.3. Statistical analysis

We performed trend analysis to observe the patterned relationship between the exposure variable (COVID-19 vaccine sentiment scores) and the outcome variables (vaccine coverage, case increase rate, and case fatality rate). We also analyzed the normal distribution of the outcome variables. We evaluated the trends of each pattern using a rank-based nonparametric test such as the Jonckheere-Terpstra test [31], which is suitable for non-normally distributed data. We then applied Spearman's rank correlation test to measure the correlation between sentiment scores and each outcome variable. All statistical tests were conducted in SAS software, Version 9.4 (SAS Institute Inc., Cary, NC, USA) [32] using a two-tailed test with a significance level α = 0.05. All data visualizations were performed using Tableau public version 2021.1 [33].

3. Results

We identified a total of 555,892 COVID-19 vaccine-related tweets from 173,968 active users during 180 days study period. Of those, 150,866 (5.94%) consists of mentions, 44,634 (1.76%) replies, 360,392 (14.18%) retweets, and 1,985,352 (78.13%) favourites with an average interaction rate of 2.68 which indicates a high interaction rate.

3.1. Sentiment trends towards COVID-19 vaccine

We observed most of the tweets, i.e., 374,180 (49%), were classified as neutral sentiments, while 140,798 (25%) tweets expressed negative sentiments and 140,624 (25%) were accounted for positive sentiments toward the COVID-19 vaccine. Fig. 2 shows the daily distribution of tweets by positive and negative sentiments throughout the study period, with the key major events annotated in the figure.

Fig. 2.

Distribution of COVID-19 vaccine-related tweets classified by sentiment during 180 days study period, with key major events annotated.

We can see the sentiments distribution was divided into two critical stages. During the first stage of the study, we observed a fluctuating trend of each sentiment, dominated mainly by negative sentiments. The negative sentiments outnumbered positive sentiments for 59 days (65.50%). However, after the vaccination rollout, the positive sentiments outnumbered negative sentiments for 56 days (62.20%). The positive sentiments trend was slightly increased on November 18, 2020 (Fig. 2A), when Indonesian President Joko Widodo stated his willingness to be the first recipient of COVID-19 vaccination [27]. The positive trend continued to grow progressively on December 7, which was associated with the arrival of the 1.2 M Sinovac COVID-19 vaccine in Indonesia (Fig. 2B).[5] The negative sentiments immediately decreased after the government decided to provide free of charge COVID-19 vaccine (Fig. 2C). The positive sentiments gradually declining but immediately increased after the National Agency of Drug and Food Control (NADFC), known as BPOM, announced that Sinovac was safe with an efficacy rate of 65.3% (Fig. 2D and 2E) [27].

However, in January 2021, the negative sentiments escalated dramatically until January 13, after President Jokowi received the first vaccine shot (Fig. 2F). This particular event has become a massive topic of discussion on Twitter because the government decided to provide vaccine shots to celebrities and social media influencers ahead of the healthcare workers. Yet, just a few hours after the vaccine shot, the celebrity was spotted violating the health protocol, which triggered many criticisms over the government vaccine campaign [27]. The negative sentiments reached their peak with 6813 tweets (Fig. 2F). However, we then observed a steady decline of negative sentiments until mid of February.

The positive sentiments displayed their peaks with 7718 tweets (Fig. 2I) after the second stage of the vaccination program began at the end of February. The NADFC also reported the safe use of AstraZeneca COVID-19 vaccines (Fig. 2K). However, the negative sentiments also showed a slight increase over the concern of public attention on the rising number of UK's new coronavirus variant (B.1.1.7) cases in Indonesia (Fig. 2J).

3.2. Most common hashtags and sentiment word clouds

To identify the most frequently used topics of COVID-19 vaccine-related tweets, we used a visualization representation of the most common hashtags (Supplementary Figure 1) and the sentiment word clouds (Fig. 3 A-3B). We found that the most prevalent hashtag was "#COVID-19" with 29,543 tweets, followed by "IndonesianPoliceCorpsEnforceVaccineDistribution" (Bahasa #PolriKawalDistribusiVaksin) with 11,075 tweets.

Fig. 3.

Sentiment word clouds of COVID-19 vaccine tweets (translated) (A) Before the national COVID-19 vaccination rollout. (B) After the national COVID-19 vaccination rollout.

When we performed the word clouds visualization, we eliminated all observed words related to COVID-19 (such as "COVID", "COVID-19", "corona", and "coronavirus") to identify the most meaningful topics in each sentiment group. As illustrated in Fig. 3A, before the vaccination rollout, the word clouds for the "positive group" appear to be related to the government's actions to import the Sinovac COVID-19 vaccine. Interestingly, the word "vaccine" has emerged as one of the most frequent words in the "negative group" that is likely from PDIP politician Ribka Tjiptaning who firmly stated her refusal against COVID-19 vaccination as the clinical trial III phase has not yet finished [27]. Meanwhile, after the vaccination rollout, the "positive group" word clouds appear to be related to the effectiveness of the COVID-19 vaccine. Finally, we observed some negative reference words appear upon the people's concern with the side effect following vaccination (Fig. 3B).

3.3. Emotion analysis before and after the launch of the COVID-19 vaccine

The result of the emotion analysis is presented in Fig. 4 A-4B. The most common emotion detected before the national vaccination rollout is the emotion of anticipation, with 32.10% of 22,395 eligible tweets. The emotion of trust was the second most dominant emotion (27.22%), followed by the emotion of surprise (13.46%), and fear (10.01%). Conversely, after the vaccination began, the emotion of trust and fear outpaced anticipation and ranked as the most prevalent emotion with 31.74% and 25.52%, respectively, out of 30,884 eligible tweets. Meanwhile, the emotion of anticipation decreased to 18.10%, and all other emotions were steadily expressed as the least common emotions (less than 10%).

Fig. 4.

Emotion analysis before (A) and after (B) the implementation of the national COVID-19 vaccination program in Indonesia.

3.4. Trend analysis of sentiment trends, vaccine coverage, case increase rate, and case fatality rate

The trend analysis results (90 days after national vaccination rollout) show a significant trend between the daily sentiment scores and the vaccine coverage (P<.0001). In addition, we also observed a significant trend between vaccination sentiment scores and the case increase rate (P<.0001) and case fatality rate (P<.0001).

We found a strong positive correlation between sentiment scores and the vaccination coverage (r = 0.71, P<.0001 both first and second doses). However, a strong negative correlation was also identified between vaccination sentiment scores and case increase rate (r = -070, P<.0001) and case fatality rate (r = -0.74, P<.0001), respectively. The daily trend of sentiment scores, vaccination coverage, case increase rate, and case fatality rate during 180 days study period was illustrated in Supplementary Figure 2 and Supplementary Figure 3.

4. Discussions

This work used social media sentiment analysis based on Twitter data to understand the public sentiments, emotions, and word usage on the COVID-19 vaccination rollout in Indonesia. Given the comparison between negative and positive sentiments before the COVID-19 vaccination began, overall public sentiments were dominated by negative sentiments with the most predominant emotion of anticipation. This finding indicates that people were still anticipating the government's intention to deploy the COVID-19 vaccine in the country. A recent survey involving 112,888 Indonesian online users reported that 64.8% of respondents expressed their willingness to get the COVID-19 vaccine, 27.6% expressed hesitancy, and 7.6% were resistant. Indonesian people were reluctant to receive COVID-19 vaccine shots because they were concerned about its safety and effectiveness. A study reported that vaccines' perceived safety and efficacy were strongly associated with intention to take the vaccine [34]. Moreover, given Indonesia has the largest Muslim population globally, people are worried that the vaccine is not halal (not permissible in Islam). Lack of trust and fear of the side effect was also reported from those who expressed vaccine hesitancy [7].

To gain public trust about the COVID-19 vaccine, the government announced the safety and efficacy of the Sinovac vaccine. The baseline effectiveness of the vaccine highly influenced the acceptance of a COVID-19 vaccine [35]. The Indonesian Ulama Council had also declared the Sinovac vaccine as halal. But surprisingly, we still observed the dominant-negative sentiments during the first month of the vaccination period. We noticed that the government's top-down communication took a slow pace to counter vaccine hesitancy and the emergence of the anti-vaccine movement in Indonesia. Moreover, the unpreparedness of the national health system, for example, the unreliable of the MoH vaccine recipient data, has caused many people not registered in the vaccination database, leading to public distrust [27].

To overcome this, at the end of January 2021, Indonesian Minister of Health Budi Gunadi Sadikin decided to use election voter data instead of MoH data to provide an updated vaccination database [27]. Furthermore, to speed up the COVID-19 vaccine rollout, the government has also recruited celebrities and social media influencers among the priority for coronavirus vaccines in Indonesia. This decision was part of the government communication strategy to encourage large-scale online followers because Indonesians are among the top global social media users [24]. With the second stage of the vaccination program targeting religious leaders, public officers, and educational personnel, kicked off in February, supported with the successful completion of the first stage of the vaccination program, the public sentiments gradually shifted to positive sentiments.

This positive emotional appeal revealed a predominant emotion of trust, associated with the proactive risk communication through social media campaigns accompanied by a robust health system and immediate actions to clarify misinformation. Given the impact of the COVID-19 pandemic, the conspiracy beliefs and trust in conventional media were reported to be the vital determinants of vaccine acceptance [36]. The increasing level of positive vaccination sentiment with the dominant emotion of trust may reinforce the acceptance of the COVID-19 vaccine [37]. Meanwhile, the negative sentiments were mainly related to fear, as people worried about the side effect after getting vaccinated. Our finding was consistent with the recent study involving Australian Twitter users who expressed fear as the top negative emotions toward the COVID-19 vaccine [38]. In addition, negative sentiments appeared upon people concerned about vaccine safety [39] and vaccine hesitancy [34]. WHO had declared vaccine hesitancy as one of the leading global health threats [40]. Thus, if people expressed fear of rejection to receive the vaccine, it is plausible that Indonesia will face a prolonged pandemic, potentially contributing to the emergence of new COVID-19 variants that may reduce the efficacy of existing vaccines and increase the risk of infections and deaths [1].

Our findings implied that the public online COVID-19 vaccination sentiment trend is significantly correlated with the increasing trend of vaccination coverage and the decreasing trend in COVID-19 case increase and case fatality rate. These results were similar to the previous studies that implied a strong and positive correlation between vaccination sentiment score expressed on Twitter and vaccine uptake [19,21,39]. Future studies should consider performing the social network analysis to explore the interactions between organizational and individual accounts with the most shared tweets or images affecting the public sentiments on vaccine uptake. Moreover, to speed up the vaccination coverage, we need to continue exploring the public sentiment on the COVID-19 vaccine over a longer period and analyzing the dynamic of public reactions, specifically toward a specific brand of vaccines and its correlation with the individual willingness to get vaccinated.

5. Limitations

Our study highlights several limitations. First, we limited our findings to specific settings (e.g., pandemic situation, particular country) or datasets (e.g., young adults as most Twitter users in Indonesia). Thus, it will decrease the generalizability of the findings to the general population and the global situation. Second, we were unable to confirm the geographical location of each tweet, so we cannot perform a clustering analysis to identify the sentiment distribution of COVID-19 vaccine-related tweets by the province in Indonesia. Third, we analyzed the public sentiments during the earlier phase of COVID-19 vaccination implementation in Indonesia (the first three months). More extended observation may capture more recent information on COVID-19 vaccination, which potentially affects the overall predominant emotions and sentiments trend. Lastly, although it is permissible to reuse public datasets on Twitter, we are concerned about the ethical issues of acquiring individual opinions without obtaining user's informed consent. However, suppose we reviewed the ethical framework of using Social Media Data [41]. In that case, we could reproduce tweets in academic publications as long as we can ensure the user's anonymity and protect their sensitive personal information. Thus, we will not violate the ethics of using social media data in research.

6. Conclusion

In conclusion, this study highlights the utility of social media sentiment analysis based on Twitter data to identify the correlation between online public sentiments on COVID-19 vaccine and vaccination coverage in Indonesia. Our study found that the public sentiments on COVID-19 vaccine-related tweets have gradually shifted from negative to positive due to the government's proactive risk communication and immediate actions to clarify misinformation surrounding the COVID-19 vaccine.

At the beginning of the national vaccination rollout, tweets were dominated by negative sentiments over positive sentiments with the predominant emotion of anticipation. However, the trend gradually changes to positive sentiments with the growing of trust emotion, which is consistent with the positive appeals of the recent news about COVID-19 vaccine safety and the government's proactive risk communication. The rising trend of the emotion of fear was quite concerning as it can induce people's willingness to receive the vaccine, putting a risk to the national vaccination uptake. We also observed a significant increase in the vaccination sentiment trend, which strongly correlates with the vaccination coverage and a strong negative correlation with the rate of case increase and case fatality during a pandemic situation.

The finding of this study suggests that social media sentiment analysis can facilitate real-time monitoring of online COVID-19 vaccine-related issues so that the governments can take proactive measures to enhance vaccine uptake and address misinformation and inappropriate behaviors on COVID-19 vaccination. Our study leads to developing effective communication strategies through collaborative efforts of public health officers, policymakers, and media sources to build public trust and disseminate credible information on COVID-19 vaccine-related issues to improve national vaccine coverage in a pandemic situation. We believe that this finding will be helpful for countries to identify and develop strategies for speed up the vaccination rate by monitoring the dynamic netizens' reactions and expression in social media, especially Twitter, using sentiment analysis.

Funding

This research was funded by the Ministry of Science and Technology (grant number: MOST 110-2320-B-038-029-MY3, 110-2221-E-038-002-MY2, and 110-2622-E-038-003-CC1) and the Higher Education Sprout Project by the Ministry of Education (MOE) in Taiwan.

Declaration of Competing Interests

The authors declare no conflicts of interest in this paper.