Beyond the hashtag: describing and understanding the full impact of the #BJSConnect tweet chat May 2019

Abstract

Background

Twitter engagement between surgeons provides opportunities for international discussion of research and clinical practice. Understanding how surgical tweet chats work is important at a time when increasing reliance is being placed on virtual engagement because of the COVID-19 pandemic.

Methods

Individual tweets from the May 2019 #BJSConnect tweet chat were extracted using NodeXL, complemented by Twitter searches in an internet browser to identify responses that had not used the hashtag. Aggregate estimates of tweet views were obtained from a third-party social media tool (Twitonomy) and compared with official Twitter Analytics measurements.

Results

In total 37 Twitter accounts posted 248 tweets or replies relating to the tweet chat. A further 110 accounts disseminated the tweets via retweeting. Only 58.5 per cent of these tweets and 35 per cent of the tweeters were identified through a search for the #BJSConnect hashtag. The rest were identified by searching for replies (61), quoting tweets (20), and posts by @BJSurgery that used the hashtag but did not appear in the Twitter search (22). Studying all tweets revealed complex branching discussions that went beyond the discussed paper’s findings. Third-party estimates of potential reach of the tweet chat were greatly exaggerated.

Conclusion

Understanding the extent of the discussion generated by the #BJSConnect tweet chat required looking beyond the hashtag to identify replies and other responses, which was time-consuming. Estimates of reach using a third-party tool were unreliable.

This analysis of the #BJSConnect tweet chat in May 2019 looks beyond the hashtag to identify the extent of dialogue between contributors. Adding in tweets and replies that did not use the hashtag almost tripled the number of contributors of original content, and identified over 100 additional tweets, many of which added to the understanding of the research paper being discussed. The findings illustrate the complexity of understanding social media dialogue and are particularly relevant, as much of our scientific and clinical discussion has moved online because of the Covid-19 pandemic.

#beyond the hasht tag

Introduction

Scientific and clinical work relies on close collaboration between peers, across disciplines, and learning from colleagues across the world. However, the COVID-19 pandemic has resulted in cancellation of physical meetings both locally and internationally throughout 2020. Understanding how virtual discussions work on social media is therefore a priority. A tweet chat is an open discussion on Twitter that is usually structured around a number of predefined questions. Participants from around the world can contribute, using the various features of Twitter, for example embedding images, sharing links, mentioning other Twitter users, replying and sharing, and quoting other tweets. Ideally, every tweet and reply in a tweet chat should include the relevant hashtag, for example #BJSConnect. However, it is easy to forget to use the hashtag as in verbal conversation individuals do not typically keep referring to the topic or individuals they are talking about.

Tweet chats involve a considerable amount of work: recruiting ‘speakers’, choosing questions, producing infographics, disseminating information about the ‘event’, and collecting information on reach and impact after the event. For the participants, social media activity often takes place out of hours, so called ‘shadow work’¹, which eats into personal and family time. A tweet chat itself requires chairing and direction just like any meeting.

During 2018 the surgical community started to organize and study social media activity in a coordinated fashion. The #SoMe4Surgery hashtag, launched on 28 July 2018, has brought clinicians together from across the world, and has catalysed surgical social media research activity, for example around the European Society of Surgical Oncology (ESSO) conference in Budapest, October 2018 (ESSO38)². The large number of tweeting participants in global movements such as #SoMe4Surgery³ and international conferences makes for a dense social network map that can be difficult to decipher visually. Unpicking a tweet chat, with a more manageable number of participants and tweets, is a more realistic prospect.

Social network analysis usually focuses specifically on tweets that used the hashtag. The present study aimed to look also at the wider activity, adding in both the replies that did not use the hashtag, and tweets quoting other tweets that had used the hashtag. By including these additional posts, the aim was to capture the social media conversation more fully in order to understand the wider reach of social media activity, and how that might apply to larger events such as conferences and health awareness campaigns.

Methods

This paper used a #BJSConnect tweet chat (20.00 to 21.30 hours GMT, 23 May 2019, scheduled to maximize potential international audience). The tweet chat featured a discussion on the age of patients undergoing surgery with Rupert Pearse (@rupert_pearse), Alexander Fowler (@buzzlogic, tweeting as @_alexfowler since 27 May 2019) and Tom Abbott (@_tomabbott), focusing on a BJS paper with Open Access for the period around the tweet chat⁴. The tweet chat was hosted by BJS Associate Editor Rebecca Grossman (@rebgross). The specific tweet chat was chosen because there had been a particular focus on extending the audience, with the journal Twitter account (@bjsurgery) sharing a social network map from a previous tweet chat earlier in the day⁵.

Eight days of tweets using #BJSConnect hashtag were extracted using NodeXL⁶ and Twitonomy⁷ the evening after the #BJSConnect tweet chat. Individual tweets, retweets, and participants were extracted from NodeXL data using methods described previously, to document tweeters, retweeters, and accounts that were mentioned in posts². NodeXL captures tweets that included the hashtag. Initial analysis of this extract involved identifying accounts that had tweeted, retweeted, and/or were mentioned in tweets. These tweets were captured in a Wakelet summary, ordered chronologically⁸. Twitonomy was used because it produces automated reports rapidly and has built in geographical mapping; it also estimates the number of times a hashtag has been viewed (impressions), using the same assumptions as medical third-party tools such as Symplur.

Additional tweets relating to the #BJSConnect conversation were identified by three methods, recording the unique Twitter identifier (URL including a 19-digit number) for each tweet. First, all tweets using the #BJSConnect hashtag from the stated period were displayed in an internet browser and the tweets with replies viewed individually (replies up to 31 May 2019 were recorded on 16 June 2019). Second, the URLs for all tweets using the #BJSConnect hashtag were entered into the Twitter search box, to identify tweets quoting these tweets. The quoting tweets were added if they had not already been identified as using the hashtag. Finally, @bjsurgery Twitter Analytics⁹ data identified additional tweets posted from the journal’s account, each of which had included the hashtag, and each of which were posted during the period of peak activity during the tweet chat (from question 6 onwards). These tweets did not appear in the original NodeXL extract, and neither did they turn up in a Twitter search for the hashtag.

These additional tweets were also added to the Wakelet summary⁸. To explore the conversations for the eight questions posted by @bjsurgery during the tweet chat, it was also necessary to follow the trail of tweets, replies, and quoting tweets. The tweets were organized in the sequence of replies.

Twitonomy provided estimates of reach, audience, and a geographical map of the location of tweeters and retweeters based on information in their user profiles and mapping via Google Maps. Estimates of ‘impressions’ (number of times a tweet was displayed on a Twitter-enabled device) using Twitonomy (third party) were compared with official Twitter Analytics statistics⁹. Twitter Analytics figures for the @bjsurgery account for 23 May 2019 were accessed and downloaded.

Results

NodeXL captured data on tweeters, retweeters, and Twitter accounts mentioned in tweets¹⁰. Fig. 1 shows the nature of contributions for the tweets that used #BJSConnect. In total, 13 accounts posted tweets using the #BJSConnect hashtag. The majority of accounts captured in the NodeXL map were either ‘just retweeters’ (48.0 per cent) or ‘just mentioned’ (34.9 per cent). Ninety-nine accounts posted original content or shared that content through retweets. Twitonomy found the same number of users in a search for the hashtag¹¹.

Fig. 1.

#BJSConnect influencers for 16-24 May 2019: just tweets using hashtag

Source: NodeXL¹⁰.

Fig. 2 shows the equivalent results after adding in the 103 tweets that replied to (61) or quoted (20) tweets including the #BJSConnect hashtag, plus the tweets posted by @bjsurgery that were invisible to Twitter searches (22). This analysis identified 102 further Twitter accounts that tweeted, retweeted and/or were mentioned. The biggest categories were again ‘just mentioned’ (42.1 per cent) and ‘just retweeted’ (32.3 per cent). There were 37 accounts identified to have posted tweets, and a total of 147 accounts that tweeted and/or retweeted.

Fig. 2.

#BJSConnect influencers, including replies and ‘quoting tweets’ relating to #BJSConnect tweets posted 16–24 May 2019

Source: NodeXL¹⁰, Twitter Analytics⁹, and hand searching of Twitter outputs.

The earliest tweet, posted a week before the tweet chat, announced the event, using the hashtag. This tweet was shared in a variety of ways during 16 and 17 May 2019, including retweets, replies that listed the names of individuals and organizations that might be interested, and tweets that quoted the original message. The next tweets occurred from 21 to 24 May 2019, with the majority (182, 73.4 per cent of total tweets in this analysis) during the 90 min of the tweet chat. The final tweet identified for this analysis was posted on 31 May 2019, replying to a tweet from the tweet chat. The tweets posted during the period of the tweet chat itself performed aspects of a critical appraisal of the paper, and explored implications and ideas for further research.

There were eight questions posted from the @bjsurgery account during the tweet chat. The number of tweets and tweeters for the questions are shown in Table 1. For six questions the responses followed an exclusively or primarily linear sequence (questions 1, 3, 5–8). For two questions, however, the responses followed a more complex structure, with key contributors making responses across different branches of the discussion. The branching responses to questions 2 and 4 are shown in Figs 3 and 4 respectively. Janet Martin’s tweets (@janetmartinwho) for question 2 catalysed considerable further dialogue, with two of her responses generating three-branched discussions (tweets highlighted by dotted lines). Additionally, Dr Martin was involved in all the different strands of conversation within this question. Question 4 also had a branching structure, but also a continuous chain of 19 responses, with further tweets branching off, from a total of 10 contributors, which started from a post on registries. Although the themes in most of the questions followed the structure and content of the paper, questions 2 and 4 covered considerably wider topics: dialogue on shared decision-making, and age versus frailty in surgical patients for question 2; and wider surgical populations and the effect of age and sex, the changing nature of surgery, including more conservative and non-operative options, for question 4.

Table 1.

Questions posted during BJSConnect tweet chat May 2019

	No. of responses	No. of tweeters responding
Q1 Why did you choose to do this study?	6	4
Q2 Why does this study matter?	36	10
Q3 How did you go about doing this ecological study?	3	1
Q4 Where are the data from?	45	11
Q5 Please tell us about the statistics	3	1
Q6 What did you find?	13	5
Q7 What are the limitations to this kind of study?	4	1
Q8 What are the implications of this study?	13	4

Fig. 3.

Flow of discussion in question 2 of the May 2019 BJSConnect tweet chat

Source: Wakelet summary⁸ of tweets identified in the full analysis and hand searching of Twitter.

Fig. 4.

Flow of discussion in question 4 of the May 2019 BJSConnect tweet chat

Source: Wakelet summary⁸ of tweets identified in the full analysis and hand searching of Twitter.

The geographical locations of 52 tweeters and retweeters were mapped using the third-party Twitter analysis website Twitonomy¹². The remaining tweeters did not have mappable information in their user profile. There were 18 countries represented across four continents, and some tweeters using more general identifiers: worldwide, international, Europe or North America. These tweets without a specific location were either pinned in the middle of the USA (international and worldwide) or the Baltic Sea (Europe). A retweeter from Barcelona (@ruthbc93) was incorrectly pinned to a district of Tulsa, Oklahoma, with the same name. Twitonomy also estimates potential reach, the maximum number of views of #BJSConnect tweets, as 4.5 million. However, as this adds the maximum number of followers for each tweet and retweet, this is likely to be a very considerable overestimate.

The third-party estimates provided by Twitonomy can be compared with official measurements direct from Twitter via Twitter Analytics⁹. There were 19 tweets posted from the @bjsurgery account during the period studied that were visible during the tweet chat. At that point @bjsurgery had 20 624 followers. Using Twitonomy estimates, these 19 tweets would have received 391 856 impressions (times that a tweet might have been displayed on a Twitter-enabled device) based on the number of @bjsurgery followers. Twitonomy and other third-party tools also incorporate retweets into their estimates, adding the number of followers for each retweeter, for each retweet. This brings the total number of estimated impressions for @bjsurgery tweets, using third-party tools, to 837 495. However, information direct from Twitter Analytics, measuring both tweets and retweets, shows that these tweets actually achieved 41 399 impressions (median 1430, i.q.r. 1107–2764). Third-party applications would therefore have overestimated impressions 20-fold for @bjsurgery’s visible tweets during this tweet chat.

Nine @bjsurgery ordinary tweets (not replies, not quoting other tweets) were posted during the tweet chat period, which were visible in Twitter search. Twitter Analytics data recorded 20 226 impressions, 33 retweets, and nine replies for these nine tweets. Based on these tweets, the @bjsurgery Twitter account received 16 retweets and 4.4 replies per 10 000 impressions during the tweet chat. These results provide estimates of the wide reach required on Twitter to achieve interactions (retweets) and new content (replies), even during a period when the potential audience would be expected to be more engaged owing to the previously advertised tweet chat.

Discussion

This analysis has documented the participants and content from the May 2019 #BJSConnect tweet chat, looking beyond just those tweets that included the #BJSConnect hashtag. It identifies a much wider network of participation than initially suggested when studying only tweets that used the hashtag. The analysis also identifies the very considerable overestimates of potential reach (measured by impressions) provided by third-party social media analytics tools such as Twitonomy, complementing previous work on reach and impact on social media¹³. These findings provide a much more comprehensive and realistic understanding of the potential role of Twitter in disseminating and discussing clinical research than previous studies.

Twitter’s main functions at an individual-user level include broadcasting and interacting, both of which have potential uses in dissemination and peer review of clinical and scientific research. In its broadcasting function, information posted on Twitter leaves a potentially permanent record that retains its quality as it is shared around the world. There has not been a technology like Twitter before: one that shares images and text like a book, videos like a TV, but transmitted instantly and in high quality across the world, to followers and people who stumble across the content, allowing immediate interactions without hierarchy. These are seductive features, and Twitter is free and does not require specialist knowledge to use. These benefits can potentially be harnessed to disseminate surgical research across the world, promoting and complementing published peer-reviewed papers. The ability to interact with other tweeters potentially adds further power to peer review.

Although providing a potentially permanent record, the practical shelf life of a tweet is 18 min¹⁴. Furthermore, the sheer volume of tweeting in a short period during a tweet chat represents a challenge in terms of processing that information during the event itself, when fellow participants are at their most receptive. Some participants will have actively searched out tweets using the hashtag, but the present study illustrates that this method misses a substantial number of tweets.

This study also illustrates that many interactions go unnoticed; the largest single group in Fig. 2 comprised the tweeters who were mentioned but did not tweet or retweet. A similar pattern is seen in conference tweeting². Twitter accounts were mentioned for various reasons during the #bjsconnect session: referencing the sources of information by mentioning journals and institutions; promoting the tweet chat to individuals and groups potentially interested in the topic; and replying to people already involved in the tweet chat. Although it is easy to make an attempted connection using social media, and this might be expected to be the best way to attract new participants to a social media activity, real-world connections are also important. Repeated engagement through different approaches (online, in person, at meetings, by e-mail, and ultimately using webinars) was successful in recruiting a wider audience in the Scottish public health community¹⁵.

The pattern of responses to questions 2 and 4 illustrates the complexity of a tweet chat. Studying this retrospectively, with information from different approaches to searching for tweets, illustrates the value of unpicking the flow of discussion. A wide range of topics were covered, with detail going beyond the original paper, identifying potential avenues for further research and collaborations, and exploring the meaning of the findings. In this way, the tweet chat moved beyond an exchange of ideas to exploring the meaning behind the exchange; from discussion to dialogue¹⁶.

Although a tweet chat captures simultaneous strands of conversation efficiently and permanently, it can be hard to follow in real time. There is a balance between efficiency, quality, and participation. Alternatives, for example online tools such as Zoom¹⁷ which have become increasingly familiar during the COVID-19 pandemic, might be easier to follow and allow discussions to unfold at a more manageable pace. However, a webinar risks being dominated by a small number of contributors, and discussion and accompanying text chat boxes provided by these tools provide only a linear record, missing the richness of interactions documented in a normal dialogue and typically visible only to participants.

The present study helps in understanding and quantifying the potential and real audience for a tweet chat, identifying the hugely inflated estimates of impressions provided by third-party software such as Twitonomy, and the inconsistencies in mapping largely resulting from ambiguous information in an individual user’s Twitter profile. The observed interactions from the extended analysis using the Twitter Analytics data for the @BJSurgery account allow an understanding of the mechanics of recruiting participants and conducting a tweet chat. In 2006, before social media became popular, there was a ‘1 per cent rule’ on the internet: for every 100 users, 89 will just view, 10 will interact, and one will create¹⁸. In contrast, the present findings suggest that a 1 in 2000+ rule would apply for social media, illustrating that a very wide reach is necessary to achieve new content.

The analysis identifies limitations in accessing Twitter data, both through Twitter search and third-party tools such as Twitonomy and NodeXL. None of these three approaches accurately identified 22 tweets posted from the @bjsurgery Twitter account. This invisibility of a substantial number of tweets is unexplained and is likely to have had a major impact on the tweet chat. Although Twitter tends to be more transparent about the display of content than other popular social media platforms, access to the Twitter application programming interface can be interrupted during peaks of activity. All technologies have their limitations, just as a microphone failure can cause major problems at a physical meeting or lack of bandwidth can disrupt a webinar.

This study demonstrates the challenges in capturing and displaying the branching structure of Twitter discussions, including replies that did not use the hashtag and quoting tweets that require a different approach to searching. A recent book on the internal workings of Twitter notes that ‘the system of followers, @ing, and threading encourages sprawling conversations to develop from initial tweets, favouring constant interaction’¹⁹. Although individual Twitter users, in this case the journal and tweet chat participants, benefit from the functions of broadcasting and interaction, Twitter harvests the data for its own use: to understand its users’ interests, refine algorithms, suggest content and people to follow, and deliver adverts. On this basis, the labyrinthine structures demonstrated in this study may be designed to keep the user on their device for longer, generating more useful information for Twitter. It is encouraging to note that in May 2020 Twitter changed both the way it displays tweets and retweets, showing more of the branching structure of responses²⁰ and displaying the quoting tweets clearly for the first time, referring to this as ‘retweeting with comment’²¹. Even with these improvements, however, describing the complex branching nature of the two questions unpicked for the May 2019 #BJSConnect tweet chat would still require considerable manual input.

There are limitations to this study. It looks at a single surgical tweet chat, run by a single surgical journal Twitter account. The findings may not be generalizable to other tweet chats or wider discussion on Twitter. However, comparison with other studies supports the findings. Tweeting at the ESSO39 surgical oncology meeting demonstrated an extensive body of tweeting and respondents ‘beyond the hashtag’, and a similar branching structure of replies and responses that also required careful manual extraction and reconstruction of data²². Tweeting at the American College of Cardiology virtual event in March 2020 (ACC.20) confirmed the considerable overestimation of potential audience using third-party tools compared with official Twitter statistics (15-fold overestimate)²³. There are, of course, differences between social media activity at conferences and tweet chats. Conferences have opportunities for real-world and structured interaction, including social activities, whereas a tweet chat or virtual event is more fragmented, with participants usually physically remote from each other.

In arranging a regular international journal club discussion of this nature, a tweet chat is arguably the simplest, most efficient, and egalitarian approach to sharing information; however, difficulties in following the chain of discussion both in real time and after the event may be barriers to participation and dissemination. A team approach to preparing, running, and analysing tweet chats is therefore recommended, establishing this well in advance of the session. For individual participants, the most important piece of advice is to use the tweet chat hashtag as this will increase the visibility of tweets and make it easier to find when reconstructing the discussion after the event.

Funding

No funding