Using the forest plot to compare citation achievements in bibliographic and meta-analysis studies since 2011 using data on PubMed Central: A retrospective study

Abstract

Background:

We saw a steady increase in the number of bibliographic studies published over the years. The reason for this rise is attributed to the better accessibility of bibliographic data and software packages that specialize in bibliographic analyses. Any difference in citation achievements between bibliographic and meta-analysis studies observed so far need to be verified. In this study, we aimed to identify the frequently observed MeSH terms in these 2 types of study and investigate whether the highlighted MeSH terms are strongly associated with one of the study types.

Methods:

By searching the PubMed Central database, 5121 articles relevant to bibliometric and meta-analysis studies were downloaded since 2011. Social network analysis was applied to highlight the major MeSH terms of quantitative and statistical methods in these 2 types of studies. MeSH terms were then individually tested for any differences in event counts over the years between study types using odds of 95% confidence intervals for comparison.

Results:

In these 2 studies, we found that the most productive countries were the United States (19.9%), followed by the United Kingdom (8.8%) and China (8.7%); the most number of articles were published in PLoS One (2.9%), Stat Med (2.5%), and Res Synth (2.4%); and the most frequently observed MeSH terms were statistics and numerical data in bibliographic studies and methods in meta-analysis. Differences were found when compared to the event counts and the citation achievements in these 2 study types.

Conclusion:

The breakthrough was made by developing a dashboard using forest plots to display the difference in event counts. The visualization of the observed MeSH terms could be replicated for future academic pursuits and applications in other disciplines using the odds of 95% confidence intervals.

Key points.

• Forest tree plots were proposed online, showing on Google Maps, which is novel and innovative in the literature.

• The social network analysis was applied to partition the MeSH terms, respectively, for bibliographic and meta-analyses.

• The ways used in this study, such as forest tree plots and social network analysis on MeSH terms, are promising and worth being applied to similar studies in the future.

1. Introduction

The field of bibliographic studies has grown exponentially over the years.^[1] An increase was seen over time in the number of articles (7410) related to the keyword bibliometric [MeSH Major Topic]^[2] on PubMed Central leading up to December 2021. Meta-analysis (MA) and systematic reviews (SRs) were compared, with 5863 and 1998 articles, respectively.^[3,4] The reason behind the growing prevalence of bibliographic and meta-analysis (or systematic literature review) studies could be attributed to the accessibility of online data and software packages specialized for analyses.^[5]

1.1. Understanding the feature of the bibliographic study

Bibliographic articles place more efforts on identifying clusters of topics, literature gaps, and academic silos, as well as the most impactful authors and their research. In contrast to meta-analysis and systematic literature review studies, bibliographic literature reviews use more quantitative and statistical methods to achieve this goal.^[5] We are, therefore, motivated to investigate whether bibliographic studies really have stronger quantitative and statistical bases compared to meta-analysis.

1.2. Using MeSH terms to compare contents in 2 types of articles

The biggest challenges are how to define the features of quantitative and statistical components in an article and how to compare them using visual presentations.

1.2.1. MeSH terms.

National Library of Medicine developed the medical subject headings (MeSH terms) as an efficient way of accessing and organizing biomedical information by reducing the ambiguity inherent from free-text data.^[6,7] MeSH terms are labels assigned to each article in Medline to describe the relevance of an article.^[8] As such, the MeSH terms can be attributed to a paper that can be considered as references to a body of knowledge stored as documents in a database.^[9]

The National Library of Medicine employees trained as “indexers” look at each new article added to Medline and assign 10 to 12 labels, which best describe the content.^[10] Furthermore, the PubMed interface is available to anyone with an Internet connection to search using MeSH terms,^[8] distinctly different from the cited references provided by the authors themselves. Thus, we attempted to investigate which MeSH terms are most associated with bibliographic and meta-analysis studies, followed by verifying whether these terms are related to quantitative and statistical methods characterizing bibliographic or meta-analysis studies.

1.2.2. Forest plot.

A forest plot^[11] has been frequently used in meta-analysis studies comparing the event counts in 2 treatments (groups), along with the overall results.^[12] Whether the forest plot can be applied to graphically compare event counts in 2 study types is worth a discussion.

1.3. Verifying quantitative and statistical methods in bibliographic studies

Over 240 articles with the keyword of 100 top-cited in the title were available on PubMed Central (PMC).^[13] Most of them simply summarized the content of the studies of a particular research field, including descriptive statistics (DS), research achievements (RAs) in countries/institutes/authors, and major topics [or research domains (RDs)]. Examples are illustrated in these articles: the global perianal fistula research,^[14] a study using the national health insurance database,^[15] one regarding atrial fibrillation,^[16] and a research reporting trends and characteristics of oral lichen planus.^[17]

A description and summary of “simple” bibliographic data (e.g., authors, journal names) is too superficial to give specific answers to desired research questions.^[5] Many bibliographic studies displayed sophisticated tables, figures, and graphs derived from bibliometric software tools and led to an overly descriptive and confusing picture of the field.^[5] Accordingly, we are going to advance the techniques used in bibliographic studies and demonstrate an approach using social network analysis (SNA)^[18] to highlight the distinct features of bibliographic and meta-analysis studies and examine the association between entities that are supported by our bibliographic data and bibliometric analyses.

1.4. Study objectives

We aimed to identify the frequently observed MeSH terms in these bibliographic and meta-analysis studies and investigate whether the highlighted MeSH terms are strongly associated, respectively, with the 2 types of study.

2. Methods

2.1. Data sources

A search was conducted on PMC using the keywords [(meta-analysis [MeSH Major Topic]) or (bibliometric [MeSH Major Topic])], where 5121 eligible articles from 2011 to 2020 were gathered on June 3, 2020. An author-made Microsoft Excel Visual Basic for Application (VBA) module was used to analyze and present the research results. All downloaded abstracts (5121) met the requirement for the type of journal article; see Supplemental Digital Content 1, http://links.lww.com/MD/G859. All data used in this study were downloaded from PMC, therefore in accordance with the regulation promulgated by the Taiwan Ministry of Health and Welfare ethical approval is not necessary for the study,

2.2. Data arrangement

With MeSH terms used to define the article types (i.e., either bibliographic, meta-analysis studies, or both), we tabulated the association between origin countries/journals and article types on publications and citations, respectively. As such, the most productive entities (e.g., countries or journals) on article types were identified.

2.3. Visual presentations on dashboards

Three types of visual presentations were provided in this study, including the pyramid plot displaying the publications and citations for article types over time, the SNA^[18] clustering MeSH terms with the most frequently observed event counts in the article types, and the forest plot presenting the results of the difference of event-count in probabilities (see the next section) between article types using the 95% confidence intervals (CIs).

2.4. Comparison of the difference in event counts between studies using forest plots

The forest plot^[11] was used to display the estimated results from paired observations and events occurred more frequently in one particular study, along with the overall results.^[12] The meta-analytical techniques have been applied in numerous observational studies (e.g., environmental epidemiology).^[19]

The forest plots were displayed in chronological order. The column on the right is a plot, including squares representing the measure of effect [e.g., odds ratio (OR)] for each of these observed studies (i.e., bibliographic and meta-analysis) and horizontal lines representing the CIs. The area of the square is proportional to the study’s weight (i.e., the sample size). The overall measure of effect is represented on the plot with a diamond. The lateral points of which indicate the confidence interval for this overall estimate.

A vertical line represents no effect(e.g., OR = 1). If the CI for an individual study overlapped with this line, it means the effect size dose not differ from no effect for the individual study at a given level of confidence(e.g., P < .05). The same scenario can be applied to the overall effect if the points of the diamond touch the line of no effect, indicating the overall result cannot be said to differ from no effect at a given level of confidence. We incorporated the forest plot on a dashboard to better present the effect on each of these observed studies with the extra function of Zoom-in and Zoom-out on Google Maps.

The power, defining how meaningful a study data set is, is denoted by the size (weight) of the box. More meaningful data tend to be those from studies with greater sample sizes and smaller CIs and have a greater contribution to the pooled result.

The forest plot is able to present the degree to which data from multiple studies observing the same effect overlap with one another.^[20] Results that fail to overlap (or fit) well are given the term heterogeneity of the data, which are deemed less conclusive. Otherwise, the data are said to be homogeneous and more conclusive. The heterogeneity is indicated by the I-square statistics^[21]; see Eqs. from (1) to (6) below.

$${\displaystyle \begin{array}{l}{\displaystyle \mathrm{I}\text{-}\mathrm{s}\mathrm{q}\mathrm{u}\mathrm{a}\mathrm{r}\mathrm{e}=100\mathrm{\%}\times \frac{Q-\mathrm{d}\mathrm{f}}{Q},}\end{array}}$$ (1)

$${\displaystyle \begin{array}{l}{\displaystyle Q={\displaystyle \underset{i=1}{{\displaystyle \sum ^k}}}{W}_i{(Y_i-{\displaystyle \overline{Y}})}^2,}\end{array}}$$ (2)

$${\displaystyle \begin{array}{l}{\displaystyle {\displaystyle \overline{Y}}=\frac{{\sum }_{i=1}^k{W}_i{Y}_i}{{\sum }_{i=1}^k{W}_i},}\end{array}}$$ (3)

$${\displaystyle \begin{array}{l}{\displaystyle \mathrm{S}{\mathrm{E}}_i={\displaystyle \underset{j=1}{{\displaystyle \sum ^4}}}\frac{1}{n_{ij}},}\end{array}}$$ (4)

$${\displaystyle \begin{array}{l}{\displaystyle \mathrm{V}\mathrm{a}\mathrm{r}\mathrm{i}=\mathrm{S}{{\mathrm{E}}_i}^2,}\end{array}}$$ (5)

$${\displaystyle \begin{array}{l}{\displaystyle W_i=\frac{1}{\mathrm{V}\mathrm{a}{\mathrm{r}}_i}.}\end{array}}$$ (6)

The df is the degree of freedom (i.e., k − 1), k represents the number of studies, n denotes the sample size (i.e., the even counts and the total observations) in 2 treatment groups,^[22] and $\mathrm{S}{\mathrm{E}}_{\mathrm{i}}=\sqrt{\mathrm{V}\mathrm{a}{\mathrm{r}}_i}$derived from Eq. (5). Var_i is the within-study variance on study i.

The computation of ORs and their CIs are addressed in Eqs. (7) to (11), where the even and noneven numbers for 2 treatment groups (i.e., 1 and 2) were set as {n1e, n1n} and{n2e, n2n} for 2 groups. Accordingly, the OR is computed by the formula of (n1e*n2n)/(n2e*n1n)^[20] in Eq. (7), and the 95% CI equal to Odds ± 1.96*SEi, see Eq. (11).

$${\displaystyle \begin{array}{l}{\displaystyle \mathrm{O}{\mathrm{R}}_i=\frac{\mathrm{E}\mathrm{v}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{C}\mathrm{o}\mathrm{u}\mathrm{n}{\mathrm{t}}_1\times \mathrm{N}\mathrm{o}\mathrm{n}\mathrm{E}\mathrm{v}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{C}\mathrm{o}\mathrm{u}\mathrm{n}{\mathrm{t}}_2}{\mathrm{E}\mathrm{v}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{C}\mathrm{o}\mathrm{u}\mathrm{n}{\mathrm{t}}_2\times \mathrm{N}\mathrm{o}\mathrm{n}\mathrm{E}\mathrm{v}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{C}\mathrm{o}\mathrm{u}\mathrm{n}{\mathrm{t}}_1},}\end{array}}$$ (7)

$${\displaystyle \begin{array}{l}{\displaystyle \mathrm{B}\mathrm{e}\mathrm{t}{\mathrm{a}}_i=\mathrm{l}\mathrm{n}\left(\mathrm{o}\mathrm{d}\mathrm{d}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{i}{\mathrm{o}}_i\right),}\end{array}}$$ (8)

$${\displaystyle \begin{array}{l}{\displaystyle Z_i=\frac{\mathrm{B}\mathrm{e}\mathrm{t}\mathrm{a}}{\mathrm{S}{\mathrm{E}}_i},}\end{array}}$$ (9)

$${\displaystyle \begin{array}{l}{\displaystyle Pro{b}_i=(1-\mathrm{N}\mathrm{O}\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{D}\mathrm{I}\mathrm{S}\mathrm{T}(\mathrm{A}\mathrm{B}\mathrm{S}(Z_i))\times 1.96,}\end{array}}$$ (10)

$${\displaystyle \begin{array}{l}{\displaystyle 95\mathrm{\%}\mathrm{C}{\mathrm{I}}_i=\mathrm{o}\mathrm{d}\mathrm{d}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{i}{\mathrm{o}}_i\pm 1.96\times \mathrm{S}{\mathrm{E}}_i.}\end{array}}$$ (11)

If all ORs in a series of studies were compared, a heterogeneity of <50% is deemed low based on I-square in Eq. (1) and indicates a greater degree of similarity between study data than an I-square value above 50%, which indicates more dissimilarity.^[11,23–25]

2.5. Creating dashboards on Google Maps

We applied the author-made modules in MS Excel and the SNA in Pajek^[26] to obtain the distinct MeSH terms to verify article types that were observed to be significantly different from each other using the forest plots. The number of citations was also analyzed and compared between article types over the years. The pages of hypertext markup language (HTML) used for Google Maps were created. All relevant bibliometric indices were linked to dashboards on Google Maps. The process of producing forest tree plots is referred to as Supplemental Digital Content 2, http://links.lww.com/MD/G860.

3. Results

The most productive countries were the United States (19.9%), followed by the United Kingdom (8.8%) and China (8.7%) (Table 1). The most number of articles were published in PLoS One (2.9%), Stat Med (2.5%), and Res Synth Methods (2.4%) (Table 2).

Table 1.

Distribution of publications across countries and article types.

Country	A	AB	B	n	%	Ci	IF
US	511	13	495	1019	19.9	7443	7.3
UK	125	3	324	452	8.8	6269	13.9
China	261	4	180	445	8.7	1261	2.8
Canada	106	3	196	305	6.0	7082	23.2
Germany	63	2	125	190	3.7	2037	10.7
Australia	71	1	84	156	3.0	1252	8.0
Spain	80	1	43	124	2.4	469	3.8
The Netherlands	38	1	75	114	2.2	1106	9.7
Italy	45	3	53	101	2.0	722	7.1
France	46		50	96	1.9	474	4.9
Brazil	74	1	13	88	1.7	216	2.5
Switzerland	26		51	77	1.5	664	8.6
Greece	13	2	55	70	1.4	2195	31.4
South Korea	38		21	59	1.2	261	4.4
India	38		10	48	0.9	117	2.4
Denmark	15		29	44	0.9	435	9.9
Iran	31		10	41	0.8	103	2.5
Japan	13		28	41	0.8	112	2.7
Taiwan	23		14	37	0.7	99	2.7
Ireland	26	1	9	36	0.7	349	9.7
Others	337	4	206	547	10.68	2770	5.1
n	2647	42	2432	5121	100.0	39,541	7.7

A = bibliometrics, AB = both, B =meta-analysis.

Table 2.

Distribution of publications across journals and article types.

Journal	A1	AB2	B3	n	%	Ci	IF
PLoS One	119	4	58	181	2.9	2647	14.6
Stat Med			158	158	2.5	1754	11.1
Res Synth Methods		2	149	151	2.4	941	6.2
J Clin Epidemiol	14	2	120	136	2.1	1866	13.7
Syst Rev	4		117	121	1.9	3019	25.0
Medicine (Baltimore)	28	3	84	115	1.8	128	1.1
Nature	87		5	92	1.5	580	6.3
BMC Med Res Methodol	2		71	73	1.2	1410	19.3
BMJ Open	23		44	67	1.1	343	5.1
BMJ	9		50	59	0.9	4222	71.6
World Neurosurg	38			38	0.6	76	2.0
Stat Methods Med Res			36	36	0.6	297	8.3
J Med Libr Assoc	20		3	23	0.4	73	3.2
JAMA	9		15	24	0.4	888	37.0
Lancet	13		10	23	0.4	126	5.5
Biometrics			24	24	0.4	207	8.6
Int J Cardiol	5		13	18	0.3	73	4.1
Arq Bras Cardiol	12		1	13	0.2	36	2.8
Int J Environ Res Public Health	21		1	22	0.3	58	2.6
Psychol Methods			22	22	0.3	219	10.0
Others	2243	31	1451	3725	76.24	20,578	4.3
n	2647	42	2432	5121	80.8	39,541	7.7

A¹ = bibliometrics, AB² = both, B³ = meta-analysis.

The meta-analysis studies earned a higher impact factor (7.1) than the bibliographic studies (5.2) (Fig. 1). The most frequently observed MeSH terms were (1) statistics and numerical data and (2) methods in bibliographic and meta-analysis studies, respectively, as displayed by the SNA (Fig. 2). The top 3 MeSH terms in each article type are linked with red lines into a triangle. Readers are invited to click on the hyperlink at the references^[27,28] to see the details laid on Google Maps.

Figure 1.

Citation analysis of publications on bibliometrics and meta-analysis.

Figure 2.

Comparison of MeSH terms frequently observed in networks.

Furthermore, 3 forest plots are presented in Figures 3 to 5. We can see that the term statistics and numerical data were statistically different over the years and in favor of bibliographic studies. The term methods also display distinctly different over the years and in favor of the meta-analysis studies, apart from the year 2020.

Figure 3.

The term of statistics and numerical data significantly observed in bibliographic studies.

Due to Figure 1 not showing the 95% CIs for difference in citation achievements between 2 article types, Figure 5 was made to help understanding that differences were present in RAs in all years but 2020 which had smaller sample size, favoring meta-analysis studies. As a result, we assigned the numbers of publication and citation to the even and noneven counts in these 2 article types, respectively, per year.

Figure 5.

Comparison of impact analysis over years between topics of meta-analysis superior to bibliographic study.

It is worth noting that only Figure 4 presents homogeneity with I-square =0 and Cochrane Q = 0.72 (P = .99). In contrast, both Figures 3 and 5 display heterogeneity with I-square = 80.99, Cochrane Q = 47.35 (P < .001) and I-square = 99.67, Cochrane Q = 2733 (P < .001), respectively.

Figure 4.

The term of methods significantly observed in meta-analysis.

4. Discussions

We applied the forest plot^[11] to compare citation achievements in bibliographic and meta-analysis studies since 2011. Two main achievements include visualizing the frequently observed MeSH terms in the 2 types of study(Fig. 2) and investigating whether the major MeSH terms of quantitative and statistical methods in a favor to the bibliographic study. In other words, the term statistics and numerical data is preferred by bibliographic study, whereas the term methods more closely associated with the meta-analysis study (Figs. 3 and 4). The higher citation achievement with meta-analysis studies was verified and shown in Figure 5.

4.1. What this knowledge adds to what we knew

We applied the forest plot^[11] with the meta-analytical techniques^[12,29] to compare the frequency of event (e.g., per MeSh term) observed in the 2 types of study. Results are shown in figures with squares and diamonds in a similar manner with the use of forest plots in meta-analysis studies, illustrating outcomes from individual samples as well as overall.^[21,30,31]

In view of the fact that SRs and MAs studies may influence and guide clinical practice,^[32] many have analyzed the quality of SRs and MAs in the literature.^[32–34] Of 126 articles in the study,^[32] 35 reviews (28%) were regarded as “poor” in terms of methodological quality, 59 (47%) as “fair,” and 32 (25%) as “good.”

Over 240 articles with the keyword of 100 top-cited in the title were searched on PMC.^[13] Most of them merely summarized the content of the studies of a particular research field reporting DS in trend, RA in countries/institutes/authors, and RD on topics.^[13–16] None were found, to date, to have assessed article quality in bibliographic fields despite the fact that much criticism has been given to bibliographic studies that only report tables, figures, and graphs derived from software tools.^[5]

4.2. What the findings imply and what should be changed?

In this study, we found that only 46% (1263/2647) of bibliographic studies and 30.9% (752/2432) of meta-analysis studies contain both the MeSH terms statistics & numerical data and methods(Figs. 3 and 4). A traditional description and summary of research data is insufficient to answer specific research questions in depth.^[5] In Figure 2, the term statistics and numerical data appears in both bibliographic and meta-analysis studies. The improvement in the quality of publications with much more MeSH terms such as statistics and numerical data is expected in the future.

Furthermore, we developed the technique of online forest plot on a dashboard that is novel and innovative in bibliographic studies and demonstrated the way using SNA^[18] to highlight the distinct MeSH terms(or topic) in a study and compare the frequency of events occurred in the 2 types of study over the years using forest plots. These features are worth replicating in other disciplines in the future.

4.3. Strengths of this study

We used SNA^[18] to analyze co-occurrence MeSH terms to identify the most frequently observed terms on a dashboard, which is different from the traditional approaches used for displaying entities on a statistic graph in the previous studies.^[35–37] We recommend that all articles should incorporate dynamical dashboards with the SNA online for audiences to look up the entities of interest on their own.

It is worth noting that we transformed the coordinates from Pajek^[26] into Google Maps so that all nodes (MeSH in this study) can be precisely located on Google Maps and all clusters can be gathered in appropriate colors and sizes on Google Maps, which were rarely seen with SNA before in literature.

Similarly, the forest plots shown on Google Maps with the 0% opacity make the background white. Online forest plot allows readers to examine the information of their own interest on the dashboard that could be useful with the zoom-in and zoom-out function superior to the traditionally statistic graphics as presented in the previous studies.^[35–37]

4.4. Limitations and suggestions

Although findings are based on the analysis above, there are still several limitations that encourage further research. First, all data were extracted from the PubMed database. Despite our effort to detect and correct, papers with incomplete information (e.g., author affiliated countries and MeSH terms) affect the results of this study.

Second, many algorithms have been used for SNA.^[18] We merely applied the algorithm of degree centrality to highlight the most influential MeSH terms in a study. Any changes in the algorithm used in this type of study might incur a different pattern and judgment to the results.

Third, the results concluded from the data extracted from PMC cannot be generalized to other databases, such as Web of Science, Scopus, or Google Scholars. However, similar approaches can be applied to other library databases in the future.

Fourth, although Figure 4 presents homogeneity among comparative occasions over the years, Figures 3 and 5 display substantially heterogeneous in data variances. The interpretation of overall random effects should be carefully made with this type of data presentation.

Fifth, we took an extra step to discuss the descriptive statistics (DS) in tables and the RA in Figure 1. For instance, the journal of Plos One has more publications on the bibliographic study (119, 67%), whereas meta-analysis studies (84, 75%) are the majority in Medicine (Baltimore). Displaying RD with visualization of article features online is required using the forest plot^[11] and applying more examples (e.g., Figs. 3 to 5) to readers in the future relevant studies.

5. Conclusion

We made a breakthrough to develop a dashboard using forest plots to display the difference in event counts over the years in 2 study types (i.e., bibliometric analysis and meta-analysis). The visualization of the observed MeSH terms illustrated in SNA could be applied to future academic pursuits and applications in other disciplines using the 95% CIs for comparison in the future.

Author contributions

J.W. developed the study concept and design. Y.C., H.Y., and W.C. analyzed and interpreted the data. L.Y. monitored the process of this study and helped in responding to the reviewers’ advice and comments. T.W.C. drafted the manuscript, and all authors provided critical revisions for important intellectual content. The study was supervised by L.Y. All authors read and approved the final manuscript.

Acknowledgments

We thank Enago (www.enago.tw) for the English language review of this manuscript. All authors declare no conflicts of interest.