Tracking the COVID-19 outbreak in India through Twitter: Opportunities for social media based global pandemic surveillance

Abstract

We investigated the utility of Twitter for conducting multi-faceted geolocation-centric pandemic surveillance, using India as an example. We collected over 4 million COVID19-related tweets related to the Indian outbreak between January and July 2021. We geolocated the tweets, applied natural language processing to characterize the tweets (eg., identifying symptoms and emotions), and compared tweet volumes with the numbers of confirmed COVID-19 cases. Tweet numbers closely mirrored the outbreak, with the 7-day average strongly correlated with confirmed COVID-19 cases nationally (Spearman r=0.944; p=0.001), and also at the state level (Spearman r=0.84, p=0.0003). Fatigue, Dyspnea and Cough were the top symptoms detected, while there was a significant increase in the proportion of tweets expressing negative emotions (eg., fear and sadness). The surge in COVID-19 tweets was followed by increased number of posts expressing concern about black fungus and oxygen supply. Our study illustrates the potential of social media for multi-faceted pandemic surveillance.

Introduction

The global outbreak of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), commonly referred to as COVID-19, has been one amongst the worst pandemics known the World history¹, resulting in adverse social, political and economic consequences around the globe². Due to the unprecedented nature of this pandemic, traditional public health surveillance methods struggled to tackle this scenario, and different strategies were adopted by different governments around the world for conducting surveillance^3–5. As of August 2021, COVID-19 outbreaks continues globally, with the highly-infectious B.1.617.2 (delta) variant being the driving force behind the waves of outbreak at the time^6–8. This variant, which was first detected in India, ravaged the country resulting in the largest national lockdown across the world⁹. As of 20^th July, 2021, India reported over 31 million confirmed cases and over 400,000 deaths. The rapid prevalence the outbreak in India, and other countries, exposed the weaknesses of traditional surveillance systems, exhibiting that most traditional mechanisms were not designed to meet the challenges of this pandemic¹⁰. For example, national surveillance methods based on testing symptomatic people, which worked effectively for certain countries, are unrealistic for others^10,11. The latest research suggests that the vaccines developed for COVID-19 are effective for the delta variant⁶, but considering the unvaccinated population across the world, it is likely that outbreaks caused by this variant will continue around the world. Even within largely vaccinated communities, many breakthrough infections have been reported due to this variant¹². There is also a possibility of new variants of the virus emerging over time, causing future outbreaks, and perhaps at faster rates. Consequently, there is a need to identify and deploy novel surveillance methods that can complement traditional surveillance approaches¹³.

One potential resource for conducting effective, real-time surveillance of COVID-19 is social media. Currently, social media adoption and usage around the globe is at an all-time high¹⁴, and social media has a global reach, with hundreds of millions of monthly active users. Despite its potential, social media has been largely underutilized for conducting close to real-time surveillance. While social media has potential negative aspects, which have been highlighted in recent literature¹⁵, the opportunities it present have not been explored sufficiently. Due to the huge global user base of social media, and the recent advances in data-driven information management approaches, such as natural language processing (NLP) and machine learning, this resource may enable the real-time monitoring of localized outbreaks. In addition to detecting outbreaks, the knowledge generated from social media may be used for conducting syndromic surveillance and understanding population-level perceptions about the pandemic. The knowledge generated over social media have been utilized in recent research for a variety of tasks such as sentiment analysis¹⁶, pharmacovigilance¹⁷, toxicovigilance¹⁸, studying mental health-related topics¹⁹, and other health related tasks²⁰.

In this study, we explored the potential utilities of social media for close to real-time pandemic surveillance. We use the recent outbreak caused by the delta variant in India to explore the potential utilities of social media. We specifically utilize the data generated on Twitter, and apply NLP methods to explore utilities beyond simple outbreak detection. Our retrospective analyses of Twitter data illustrate that it may potentially be used for gaining near realtime insights about various aspects of the pandemic. In this paper, we discuss the following possibilities:

• Utilizing the volume of COVID-19 data generated on Twitter, combined with geolocation-related metadata, for detecting outbreaks.

• Conducting syndromic surveillance through the active monitoring of user-reported symptoms on Twitter via the use of NLP.

• Assessing the mental/psychological impacts of outbreaks via automated emotion analysis of COVID-19-related Twitter chatter.

• Detecting emerging concerns related to the pandemic in targeted populations via automated Twitter chatter analysis.

We present our methods and findings in the following subsections. We have also publicly reported our findings here publicly available via an interactive, web-based dashboard.^†

Materials and methods

Study setting

Our study is based on data generated publicly on the social network Twitter. Twitter is one of the most popular social networks in the world, with close to 400 million monthly active users in 2021²¹. Twitter is a particularly attractive resource for real-time data analysis because most of the data on this platform is public. Posts on Twitter, referred to as ‘tweets’, are essentially ‘microblogs’ and are typically publicly available. Posts on Twitter may also have associated meta-data, which can be leveraged to obtain additional information, such as geolocation-specific statistics.

Data collection

We collected data from Twitter via its streaming application programming interface (API) that was specifically created for conducting COVID-19 research²². The COVID-19 stream API was released by the company to enable researchers study the conversation surrounding COVID-19, and the authors of this manuscript were granted special permission to access the stream. Unlike the traditional public streaming API, which only provides access to a sample of tweets posted at any time, the COVID-19 API stream delivers all the conversations about the topic without any rate limitations. We collected all posts in English from this API using COVID-19-related keywords (eg., ‘COVID’, ‘COVID19’, ‘corona virus’),^‡ and utilized the metadata associated with the tweets to geolocate the sources, specifically tweets that originated from India. When available, we used the geolocation coordinates of tweets to identify specific regions of India from which each tweet originated. We also collected data about India during this period by adding a second layer of filter containing term ‘India’ to understand global response. For the experiments described in this paper, we used data collected in this manner from the beginning of 2021 until July 2021. The streaming big data was stored in a mongoDB database²³, and NLP methods were applied to derive insights from the data.

Data analyses

We conducted analysis to explore several aspects of the Indian outbreak, specifically (i) outbreak timeline and location analysis, (ii) real-time syndromic surveillance, (iii) population-level emotion analysis during the outbreak, and (iv) emerging topic detection. We described the methods applied for the specific analyses in the following subsections.

Outbreak timeline and location

We used the volume of COVID-19 related tweets over time geolocated from India to track the outbreak timeline. We compared the timeline with other events in India (eg., opening of public places such as shopping complexes and movie theatres). Whenever possible, we mapped the tweets to the states from which they were posted. We used 2 methods to detect the geolocation origins of the tweets. First, for tweets that had geolocation coordinates available, we mapped them to the specific state in India. For tweets that did not have geolocation coordinates in the meta-data, we used an existing package called geo-carmen24. This package uses the meta-data provided by the Twitter API to extract their location from their geo-tagging information, user profile etc. The geolocation information is further possibly segmented into country, state and county level. This helps in identifying the precise locations and to identify patterns specific to certain areas.

We additionally used a previously-developed machine learning classifier to detect tweets that represented self-reports of COVID-19 positive tests (ie., users who reported that they had tested positive for COVID-19)25. The classifier was trained to use posts from Twitter which mentioned COVID-19-related keywords. These posts were manually annotated to indicate self-report or otherwise. To prepare the texts of the tweets for this classifier, we had to perform some basic preprocessing on the text. We first tokenized the texts by breaking lines into words, called tokens, followed by converting all the data into lower case. We also removed stopwords, punctuations, numbers, extra spaces, and special characters using “cleantext” package. The detection of self-reports was modeled as a binary classification task, and we applied a state-of-the-art method called bidirectional encoder representations from transformers (BERT)²⁶ on manually annotated data.

In addition to tracking posts originating from India, we also tracked the global response to the Indian COVID-19 outbreak by identifying tweets emerging from outside India that contained both: COVID-19 related keywords and references to India. We geolocated tweets at the country-level to identify which countries, other than India, had high interest in the outbreak.

Syndromic surveillance

We applied a previously-developed COVID-19 Twitter symptom lexicon²⁷ to detect specific symptoms that were reported by users from India. The lexicon contains non-standard expressions and misspellings that are commonly found in social media data. To detect symptoms from the text, we applied NLP to perform inexact matching. This enabled us to detect symptom expressions that were lexically similar to those encoded in the lexicon, but not necessarily identical. The lexicon was used to map symptom expressions to standardized IDs in the Unified Medical Language System (UMLS)²⁸. We then computed the frequency of each symptom.

Emotion analysis

Our intent was to analyze the emotions expressed via tweets before, during and after the outbreak, that could also potentially detect changes in emotions over time. We performed linguistic emotion analysis of the tweets using the lexicon curated by the National Research Council, Canada, which contains a comprehensive list of approximately 14,182 English words related to anger, fear, anticipation, trust, surprise, sadness, joy, sentiment (negative and positive), and disgust²⁹. In addition to this, we also quantified the aggregated anxiety levels over time, as expressed by the tweets. We tried to identify the changes in user’s emotions pre-outbreak and post-outbreak. We considered pre-outbreak period as January and February while March, April, and May as post-outbreak period. The intensity of each emotion is measured between 0 to 1 at the tweet level, where 0 is the least and 1 to be the highest.

Detecting emerging concerns

We used frequency distributions to identify emerging concerns and interests associated with the outbreak. For detectable emerging concerns, we tracked their distribution over time by tracking frequencies of word bigrams and trigrams (Figure. 1). Bi-grams and tri-grams are collectively called n-grams, which represent contiguous sequences of n words. This led us to discover multiple topics—black fungus, a disease that widespread incidence in India following the COVID-19 outbreak, and vaccine-related chatter, specifically represented by the keywords ‘CoviShield’ and ‘CoVaxin’, which represent the two vaccines that were available in India at the time.

Figure 1.

Most frequently occurring bi-grams in our COVID-19-related Twitter data.

Results

Outbreak and location

Between January and July 2021, we collected over 4 million tweets about the outbreak in India, of which over 500,000 were geolocated to be from India, with 9,700 having specific geolocation coordinates. Globally, 3.56 million tweets were posted on India from other countries. Figure 2 presents the timeline of COVID-19 tweets geolocated which have been posted from India between early January 2021 to early June, 2021. The figure also shows the timeline of daily confirmed COVID-19 cases in India, and the timelines for two important national events—the opening of public places such as shopping centers and state-level elections. From the figure, it can be seen that the daily volume of tweets closely followed the number of confirmed COVID-19 cases. We also found a statistically significant correlation (Spearman r=0.944, P=0.001) between the 7-day moving average of tweet count and 7-day moving average of COVID cases reported.

Figure 2.

Comparison of weekly volume of COVID-19 related tweets from India (top) and the number of confirmed COVID-19 cases per day (bottom).

Figure 3 shows the state-level distribution of tweets during this timeframe. Darker shades represent higher numbers of tweets. Significant correlations for the COVID-19 cases in the states were found with the volume of Twitter data available (Spearman r = 0.84, p = 0.0003). The highest number of tweets were from the states Maharashtra (~24%), Karnataka (~11.5%), Uttar Pradesh (~7.5%) and Tamil Nadu (~7%). The correlation between tweet count and COVID-19 cases recorded for the top 4 states (Maharashtra, Karnataka, Uttar Pradesh, and Tamil Nadu) is also strong, however not statistically significant (Spearman r = 0.8, p= 0.200) due to the low number of available data points. Maharashtra, which also has one amongst the largest cities in India (Mumbai) also had the highest number of COVID-19 cases during this time, than the other states. The three other states with the highest number of tweets were among the next top 5 states in terms of highest numbers of COVID-19 cases. Kerala and Andhra Pradesh were the two other states that had high numbers of confirmed COVID-19 cases but relatively lower number of tweets. The number of self-reports detected via supervised classification was relatively low, peaking at 109 reports on April 18^th. In total, 374 COVID-19 self-reports recorded in the first half of 2021. This finding is different from the other conducted by similar recent studies similar recent studies focusing on other geolocations, which showed high numbers of self-reports during early COVID-19 outbreaks²⁷.

Figure 3.

State-level distribution of tweets about COVID-19 during our collection period (January to July 2021). Darker shades represent higher numbers of posts.

Syndromic surveillance

The most commonly discussed/reported symptom was fatigue, followed by cough and dyspnea (shortness of breath). The number of mentions about of fatigue were more than double the next highest reported symptom (cough). Other detected symptoms included headache, anosmia (loss of smell) and loss of appetite. All these symptoms were among the top 8 symptoms of acute COVID-19 that were detected to be reported by COVID-19 positive Twitter users. Relatively speaking, two symptoms that were underreported were pyrexia (fever) and body ache & pain.

Emotion analysis

Compared to the pre-outbreak time period, there was a detectable surge in negative emotions during the outbreak, namely fear, sadness, and anger (Figure 4). As much as six times the pre-outbreak count of tweets prone to fear, anger and sadness were posted during the months of April and May, which coincided with the outbreak period.

Figure 4.

Changes in negative emotions (anger, fear, sadness) detected from text during pre-and post-outbreak periods in India.

Emerging concerns

In addition to increased volume of COVID-19 tweets during the outbreak, there is a high levels of negative emotions, and high levels of anxiety expressed in the tweets, our NLP-driven analyses specifically discovered three topics that has emerged during the Indian outbreak—black fungus, oxygen supply, and COVID-19 vaccines. Many cases of mucormycosis, commonly referred to as black fungus, were detected in Asian countries, particularly India, following the outbreak of the delta variant of the COVID-19 virus³⁰. In general, the main source of mucormycosis infections is from multiple contaminated sources and tend to infect diabetic patients quickly. As the number of COVID-19 cases increased in India, the number of black fungus infected particularly among diabetic patients, also increased³¹. Chatter about black fungus originating from India started rising from early May which peaked on the 20^th of the month, approximately two months after the outbreak-related chatter surged on Twitter (Figure 5a). The timeline of the rise in black fungus chatter coincided with the extraordinarily high numbers of post-COVID infections of cerebral mucormycosis infections in the country³².

Figure 5.

(a) Volume of tweets from India mentioning black fungus over time; (b) Volume of tweets mentioning ‘CoVaxin’ or ‘CoviShield’ over time. The figure also shows the timeframe when the supply of CoVaxin was reduced.

Vaccine-related chatter also surged during this wave of outbreak in India, although, unlike black fungus, the increase in such chatter was steady from early 2021 and continued to increase during the outbreak months. Two prominent vaccines have been widely used in India—CoVaxin and CoviShield. Among the vaccine-related chatter, nearly 44% of the users discussed about CoVaxin while 56% about CoviShield. Early on in the year, CoVaxin was the most commonly discussed vaccine, but it was later surpassed by CoviShield (Figure 5b). Interestingly, the increase in CoviShield-related chatter relative to CoVaxin coincided with a reduction in supply of the latter, around late April. Another topic of interested identified through twitter chatter was oxygen supply. As the COVID-19 infections rapidly increased, India a faced crisis in oxygen supply. The requirement of oxygen was observed in the Twitter chatter as the topmost mentions about oxygen included oxygen supply, oxygen bed, oxygen concentrators, and oxygen cylinders, as shown in Figure 6.

Figure 6.

Frequently used terms with oxygen and their number of mentions in the twitter chatter.

Early detection

Our findings also suggest that Twitter may be used for predicting the outbreak of COVID-19, by detecting symptom-mentioning posts. We found a strong and significant correlation between the number of covid cases recorded and the counts of symptomatic tweets mentioned a week earlier in the chatter (Spearman r=0.897, p=0.000). Early detection/prediction may have a significant impact by enabling us to estimate future hospitalization needs.

Discussion

Social media, specifically Twitter, chatter encapsulates information in abundance regarding COVID-19. The knowledge contained within this resource can potentially be leveraged to obtain real-time insights about the current pandemic and also future pandemics. Our explorations on large-scale Twitter data generated specifically during the delta variant outbreak in India suggests that such data can be utilized to obtain multifaceted insights, adding to the detection of geolocation-specific outbreaks. While we used the Indian outbreak as our chosen topic, the methods outlined in this paper may be applied to any specific region of the world and over any social network. For our study, Twitter was a suitable social network as India has the third highest number of Twitter users in the world, following the United States and Japan³³. For our study, the COVID-19 streaming API made it possible to collect user-posted data in real time. Most social networks provide APIs that can be leveraged to obtain real-time insights, thus, for monitoring pandemics in regions with lower numbers of Twitter users, the most relevant social networks can be used.

One of the primary challenges of using social media data for obtaining such multi-faceted insights is its difficulty of mining knowledge from the noisy text-based data that is generated^34–36. The text-based knowledge that is generated is typically hidden in large volumes of noise, non-standard terminologies, misspellings, and ambiguous expressions. While it is possible to extract relevant knowledge, by manual inspection of each data point, the large volume of data makes manual curation on a continuous basis impossible, particularly in real-time. Thus, such manual curation and analyses of data are generally retrospective in nature^37,38. Importantly, manual curation is typically limited to small data samples, and relying on such curation methods takes away one of the major advantages of social media— the availability of big data. Thus, leveraging social media data optimally requires the development of advanced data science, NLP and machine learning methods, which can effectively characterize streaming data. In addition to addressing the above-mentioned challenges associated with social media data, such approaches also need to address emerging problems on social media, as in misinformation^39,40, often referred to as an infodemic¹⁵.

One among the key findings of this work is the high correlation between the COVID-19 case numbers and the volume of Twitter chatter. States in India with higher numbers of COVID-19 cases also tended to have high volumes of chatter. We observed that states with large metropolitan cities such as Mumbai (Maharashtra), Bengaluru (Karnataka), and Chennai (Tamil Nadu) tend to produce higher volumes of chatter associated with the outbreak. There are two explainable reasons behind this: (i) large metropolitan cities are generally the epicenters of outbreaks, which was no different in case of the Indian outbreak; and (ii) such metropolitan cities also have large numbers of technologically adept, young people, who make up to the vast majorities of social media users. In addition to mirroring the outbreak, the Twitter chatter also revealed symptoms reported/discussed by the users, which may be used to conduct syndromic surveillance. In areas with low number of testing locations, social media based syndromic surveillance may provide early signals about upcoming outbreaks. Interestingly, although there was a large volume of COVID-19 related chatter emerging from India, unlike prior studies conducted on populations from other countries (eg., United States), we found low number of self-reports on COVID-19 positive status. While exploring the reasons behind this is outside the scope of our work, we suspect it might be because of stigma associated with COVID-19⁴¹. It is possible that the tweet volumes were in response to the rising case numbers, and not necessarily predictive.

In addition to outbreak and syndromic surveillance, our study shows that Twitter data can be effectively utilized to assess population-level emotions associated with the pandemic. The rise in negative emotions may not only be a response to COVID-19, but also to the social distancing and ‘lockdown’ measures implemented by regional governments. A number of recent studies have elaborated on the negative mental health consequences of the pandemic^42,43, and our study suggests that social media data at the time of an outbreak maybe utilized for better understanding on geolocation-specific mental health impacts (eg., by studying/detecting expressed emotions). Interventions or social programs may be guided by population-level mental health assessments at specific times and places. While we attempted to assess emotions in a relatively simplistic manner, more sophisticated approaches under the umbrella of sentiment analysis⁴⁴ which might be employed for more targeted information. For example, sentiment analysis methods have been employed in the past to study people’s perceptions about vaccines⁴⁵ and treatments⁴⁶. A recent systematic review discussed many applications of sentiment analysis approaches during pandemics, such as the current one, and infectious disease outbreaks⁴⁷.

Strengths and limitations

The methods we described in this paper have several advantages compared to more other traditional surveillance approaches. Firstly, the social media based surveillance methods discussed are purely data-centric, and they do not incorporate hypothesis-driven biases. For emerging health crises with many unknown issues, like the current COVID-19 pandemic, data-driven approaches can provide insights on topics that may have been overlooked under other circumstances. Secondly, streaming social media based monitoring can be done at real-time or close to it. This can significantly reduce the lag times associated with many traditional surveillance approaches that rely on tools such as surveys or require compiling data from multiple sources (eg., reports from testing centers). Rapid outbreaks, such as those observed during the COVID-19 delta variant, require rapid responses, and data-centric approaches over social media data may aid such processes. And finally, the large user base of social networks means that they can potentially provide access to hard-to-reach populations. While our study is exploratory by nature, the methods we applied may be built on to establishing participatory surveillance through social media^10,48. In our work, information flow was unidirectional, but there is a potential for establishing bidirectional social media based communication channels to complement the existing public health education, surveillance and intervention methods. Such bidirectional communication models have been explored in recent literature for targeted topics, such as the use of chatbots for mental health support⁴⁹, but their utility for pandemic surveillance and response has not been focused.

Our study has several limitations as well. The primary limitation stems from the user base of Twitter—users tend to be younger than the general population and is thus not representative of the general population. However, over recent years, social media adoption has increased significantly among older demographics⁵⁰. Also, our study only focused on tweets that were in English. This limitation was introduced because the NLP tools we employed were not designed for multilingual text processing. Recent research in the field of computational linguistics has focused on developing multilingual corpora to aid multilingual NLP, and similar efforts have been reported for clinical texts⁵¹. Purely data-centric methods such as ours are also vulnerable to potential data manipulation by bots or automated accounts, and to misinformation. Both of these problems persist in all social media based studies, and currently researchers and social network administrators are actively engaged in reducing the impacts of these. Twitter, for example, has recently adopted a zero tolerance stance for accounts spreading misinformation, and actively suspends or closes such accounts.

Conclusion

Social media, specifically Twitter, chatter encapsulates an abundance of information regarding COVID-19. The knowledge contained within this resource can potentially be leveraged to obtain real-time insights about the current pandemic and may help in forecasting future pandemics. While our study focuses solely on India, the same methods can be applied to conduct real-time surveillance in other countries, including the United States. It must be noted that while we have outlined the utilities of social media for pandemic surveillance, we do not advocate the replacement of traditional methods of surveillance by social media based ones. Traditional surveillance methods, such as those relying on testing center numbers, hospital admissions, and contact tracing methods, to name a few, have been established over the years through evidence-based research. Our findings suggest that social media has high potential for complementing traditional surveillance methods, and as the user base of social media grows, the utility of such platforms may further increase in the future. Future research efforts should investigate further how social media can complement traditional surveillance methods, and also how they may be utilized for participatory surveillance and interventions.