Does a research group increase impact on the scientific community or general public discussion? Alternative metric-based evaluation

Abstract

In this study, we investigated the impact of scientific publications of the Italian SIMPAR (Study In Multidisciplinary PAin Research) group by using altmetrics, defined as nontraditional metrics constituting an alternative to more traditional citation-impact metrics, such as impact factor and H-index. By correlating traditional and alternative metrics, we attempted to verify whether publications by the SIMPAR group collectively had more impact than those performed by its individual members, either in solo publications or in publications coauthored by non-SIMPAR group investigators (which for the purpose of this study we will refer to as “individual publications”). For all the 12 members of the group analyzed (pain therapists, biologists, and pharmacologists), we created Open Researcher and Contributor ID and Impact Story accounts, and synchronized these data. Manually, we calculated the level metrics for each article by dividing the data obtained from the research community by those obtained from the public community. We analyzed 759 articles, 18 of which were published by the SIMPAR group. Altmetrics demonstrated that SIMPAR group publications were more likely to be saved (77.8% vs 45.9%), discussed (61.1% vs 1.1%, P<0.0001), and publicly viewed (11.1% vs 1.3%, P=0.05) than individual publications. These results support the importance of multidisciplinary research groups in the impact of scientific literature; the interaction and synergy among the research participants allowed the obtainment of high impact-literature in the field of personalized pain medicine. Finally, our findings demonstrate the potential of altmetrics in estimating the value of the research products of a group.

Introduction

Altmetrics creates a new approach to evaluating the impact of publications by considering the number of downloads, shares, and discussions on social networks.¹ This approach does not replace the traditional bibliometric indicators, such as Impact Factor and H-index, but rather focuses on new aspects of publication impact.² Although still in its infancy, altmetrics has the potential to become a valid assessment strategy for the evaluation of publication impact.³ Altmetric tools capture information through the use of metrics from HTML views and downloads of articles, blog posts, tweets, bookmarks, etc. All of these sources are alternative indicators of impact that go beyond traditional citation, focusing on the content and uses of the social web,⁴ with this information provided in real time. Altmetrics elucidate not only the impact of scientific research by researchers but also the impact of the research on the public through social media.^2,5 In fact, through altmetrics, the impact of research can be measured at the individual-article level, using a combination of such data as the number of times that a particular paper has been downloaded, discussed, shared, and cited.⁶ This approach allows not only researchers but also institutions to analyze postpublication activity around a paper in near-real time using various online resources.⁵

To use this new tool, we focused on the Italian researchers of the SIMPAR group, which was founded in Pavia in 2007. This group has rapidly become an eminent translational group in the pain field, with its annual meeting now considered a major international pain conference (www.simpar.eu). In addition to the authors of this study, the other nine members of the SIMPAR group are Drs Marco Baciarello, Dario Bugada, Christian Compagnone, Andrea Fanelli, Stefano Govoni, Maurizio Marchesini, Cristina E Minella, Carolina Muscoli, and William Raffaeli. SIMPAR’s multidisciplinary collaboration has included several professionals of different disciplines and has produced a number of publications on the personalization of pain therapy through a multidisciplinary approach, including traditional medical, genetic, epigenetic, and “omic” disciplines. Table S1 lists each of the 18 papers published by at least two SIMPAR members in collaboration between 2010 and 2015. As described herein, we have been able to obtain statistically significant results regarding the force of the group as a whole in both the research and public communities.

Materials and methods

For each of the 12 researchers of which our team is comprised, we created an ORCID (Open Researcher and Contributor ID) account (www.orcid.org), in addition to an Impact Story (https://impactstory.org) account that imported our data and synchronized it with the unique ORCID identifiers. Collected items were assigned to specific categories, such as “cited” (or highly cited), “saved” (or highly saved), or “discussed”. In doing so, our Impact Story provided us with data regarding the number of times an article was saved by scholars, cited by other researchers, publicly discussed (Facebook, etc), and cited by the general public (blog posts, Wikipedia). These metrics were classified along two dimensions: audience (scholars or the public) and type of engagement with the online research products (viewed, discussed, saved, cited).⁷ From Impact Story, we were able to retrieve all altmetrics data for the 12 researcher accounts (paper citations, discussions, views by the research community or public).

Statistical analysis

Through the personal profiles of altmetrics, for each member of the SIMPAR group, we were able to count the number of citations, times a paper was saved, and discussions from the public community for each paper published. Then, we compared the SIMPAR group percentages of articles cited (or highly cited), saved (or highly saved), or discussed relative to those published by single authors (either written alone or in collaboration with coauthors who were not members of the SIMPAR group) by means of Fisher’s exact test. Quantitative variables are described as median and interquartile range (IQR), ie, the 25th and 75th percentiles and compared to collective SIMPAR data or individual publication articles by means of a nonparametric Mann–Whitney test. The association between citations from Scopus and the altmetric score or its components (eg, Facebook posts, tweets, Mendeley readers) was expressed through a nonparametric Spearman’s ρ-correlation coefficient. A P-value <0.05 was considered statistically significant. All analyses were performed utilizing Stata 14 (Statacorp LP, College Station, TX, USA).

Results

We analyzed 169 articles, 18 of which were SIMPAR group publications. The median number of articles by the authors was 17 (IQR 10–27), and the median year of publication was 2013 (IQR 2011–2014). Altmetrics demonstrated that SIMPAR group publications were more likely to be saved (78% vs 53%, P=0.05) and publicly discussed (61% vs 4%, P<0.0001) than individual publications. However, no significant difference emerged between the SIMPAR group publications and individual publications in terms of being cited (cited 44% vs 36%, highly cited 22 vs 11%; P=0.20) and publicly viewed (11% vs 3%, P=0.25). Notably, eleven of 18 articles published collectively by the SIMPAR group received a tweet (median 1, IQR 1–3), while only 36 of 151 individual publications received a tweet. Moreover, 15 articles from the SIMPAR group collectively were accessed through Mendeley readers (median times accessed 4, IQR 1–11) versus 85 of the individual publications (median times accessed 2, IQR 0–8; P=0.01). We describe the correlation between Scopus citations and the single components of the analyzed alternative metrics in Table 1. We found that the alternative metrics were generally low, with the exception of those for Mendeley readers (ρ=0.47, P<0.0001) (Figure 1).

Table 1.

Correlation coefficient (P-value) between Scopus citations and altmetric components

Source	Facebook posts	Tweets	Impact Story views	Mendeley readers
Tweets	0.02 (0.81)
Impact Story views	−0.01 (0.87)	0.13 (0.09)
Mendeley readers	−0.04 (0.6)	0.01 (0.87)	0 (0.93)
Scopus citations	−0.05 (0.53)	−0.10 (0.18)	−0.04 (0.57)	0.47 (<0.001)

Note: Data presented as correlation coefficient (P-value).

Figure 1.

Correlation between Mendeley readers and Scopus citations.

Discussion and conclusion

We found significant correlations between the SIMPAR group collective publications and their impact on the indicator linked to research activity (Mendeley readers), although not to public discussion (such as Facebook and tweets). However, the impact of collective SIMPAR group articles was high also on general public items, even though it did not reach statistical significance. This is meaningful, as funders, universities, and publishers increasingly demand indicators of the impact of science on society.⁸

Moreover, we are confident that the new metrics of medical groups could have an impact on the pain-patient community as well. Through online platforms, such as Twitter and Facebook, like-minded people can form their own communities to discuss their shared experiences, problems, and more.

Our results also illustrate how collaborative multidisciplinary teams and their projects improve the overall impact of researchers’ work on the researchers themselves. If collaborative efforts are more widely disseminated than individual publications, as our study suggests, such efforts can potentially provide additional exposure to group members, which may result in greater career enhancement than individual publications. Through these avenues, researchers can leverage social media opportunities to their own professional and academic advantage.⁹ Enhanced exposure allows for the sharing of ideas and research among respective networks, spotlighting pain studies both nationally and internationally.

Finally, as the SIMPAR group take research ethics seriously, we opine that our collaborative approach is an ethical one as well as an effective one. Research resources are scarce, and becoming more so. Ways to increase the “yield” from biomedical research have been identified as an imperative.¹⁰ As our collaborative efforts have been demonstrated to enhance access to meaningful, clinically relevant research results, our approach results in more “bang for the buck” by more readily disseminating useful information to practicing clinicians, as well as to other researchers interested in building upon the fund of data obtained through our investigations. Hopefully, other pain researchers will choose to follow our lead.

Supplementary material

The 18 papers published from Italian SIMPAR group (from at least two members) analyzed in the paper are as follows:

• Perotti L, Cusato M, Ingelmo P, et al. A comparison of differences between the systemic pharmacokinetics of levobupivacaine and ropivacaine during continuous epidural infusion: a prospective, randomized, multicenter, double-blind controlled trial. Anesth Analg. 2015;121(2):348–356.

• Catenacci SS, Lovisari F, Peng S, et al. Postoperative analgesia after laparoscopic ovarian cyst resection: double-blind multicenter randomized control trial comparing intraperitoneal nebulization and peritoneal instillation of ropivacaine. J Minim Invasive Gynecol. 2015;22(5):759–766.

• Casale R, Di Matteo M, Minella CE, Fanelli G, Allegri M. Reduction of painful area as new possible therapeutic target in post-herpetic neuropathic pain treated with 5% lidocaine medicated plaster: a case series. J Pain Res. 2014;7:353–357.

• Bugada D, Allegri M, Lavand’homme P, De Kock M, Fanelli G. Inflammation-based scores: a new method for patient-targeted strategies and improved perioperative outcome in cancer patients. Biomed Res Int. 2014;2014:142425.

• Gigliuto C, De Gregori M, Malafoglia V, et al. Pain assessment in animal models: do we need further studies? J Pain Res. 2014;7:227–236.

• Gardella B, Porru D, Allegri M, et al. Pharmacokinetic considerations for therapies used to treat interstitial cystitis. Expert Opin Drug Metab Toxicol. 2014;10(5):673–684.

• De Gregori S, Minella CE, De Gregori M, et al. Clinical pharmacokinetics of morphine and its metabolites during morphine dose titration for chronic cancer pain. Ther Drug Monit. 2014;36(3):335–344.

• Compagnone C, Tagliaferri F, Allegri M, Fanelli G. Ethical issues in pain and omics research. Some points to start the debate. Croat Med J. 2014;55(1):1–2.

• Bugada D, Guardia Nicola F, Carboni V, Allegri M. Transversus abdominis plane catheter infusions after major abdominal surgery in morbidly obese patients: reply to comments. Minerva Anestesiol. 2014;80(6):747.

• Fanelli A, Ghisi D, Allegri M. Is spinal anaesthesia a suitable technique for ultra-short outpatient procedures? Acta Biomed. 2013;84(1):76–80.

• Bugada D, Guardia Nicola F, Carboni V, Allegri M. TAP block for opioid-free postoperative analgesia in obese surgery. Minerva Anestesiol. 2013;79(12):1447–1448.

• De Gregori M, Garbin G, De Gregori S, et al. Genetic variability at COMT but not at OPRM1 and UGT2B7 loci modulates morphine analgesic response in acute postoperative pain. Eur J Clin Pharmacol. 2013;69(9):1651–1658.

• Mura E, Govoni S, Racchi M, et al. Consequences of the 118A>G polymorphism in the OPRM1 gene: translation from bench to bedside? J Pain Res. 2013;6:331–353.

• Allegri M, Clark MR, De Andrés J, Fanelli G. Pain treatment: a new approach to link bench to bedside – the SIMPAR meeting 2011. Eur J Pain. 2012;16(6): 779–781.

• De Gregori S, De Gregori M, Ranzani GN, Allegri M, Minella C, Regazzi M. Morphine metabolism, transport and brain disposition. Metab Brain Dis. 2012;27(1):1–5.

• De Gregori M, Allegri M, De Gregori S, et al. How and why to screen for CYP2D6 interindividual variability in patients under pharmacological treatments. Curr Drug Metab. 2010;11(3):276–282.

• Allegri M, De Gregori M, Niebel T, et al. Pharmacogenetics and postoperative pain: a new approach to improve acute pain management. Minerva Anestesiol. 2010;76(11):937–44.

• Allegri M, Lombardi F, Custodi VM, et al. Spontaneous cervical (C1-C2) cerebrospinal fluid leakage repaired with computed tomography-guided cervical epidural blood patch. J Pain Symptom Manage. 2010; 40(3):e9–e12. Erratum in: J Pain Symptom Manage. 2010;40(6):e9.