New quality and quantity indices in science (NewQIS): results of the first decade—project progress review

Abstract

Strategies employing information science and scientometric approaches were introduced to science policy and management over the past decades. As a rapidly evolving field, new bibliometric parameters are proposed and discussed continuously and the fields also benefits from the introduction of novel visualization techniques. The present article summarizes the experiences with a platform that combines geographical mapping with scientometrics. It was established between 2005 and 2008 at the Charité in Berlin and termed “New Quality and Quantity Indices in Science” (NewQIS), consisting of the integration of common scientometric parameters such as the h-index and novel visualization techniques including density equalizing mapping. NewQIS was used to assess socio-economic important fields of medicine and sciences. Within NewQIS studies, research activities, citation patterns and their relation to socio-economic figures were analyzed with regard to time periods, countries, continents or even single cities. Within the decade after its establishment, more than 80 NewQIS articles were peer-reviewed and published. Being a non-funded low budget project, it was used by many medical students to conduct their MD thesis. The narrow technical frame led to the chance of a comparison of research output between different fields of science. This article summarizes NewQIS 1.0 activities, discusses its limits and gives a look into the future of NewQIS 2.0 with a target of 200 evaluated entities of the biomedical field of sciences.

Introduction

Academic science is big business and big money. Billions of US-dollars (USD), Euros and other currencies are channeled into academic science every year. As a matter of fact, the decision makers—politicians and career officials—want to know about the fate of funding: Did it work? What was done? How much was done? Who did it? Who did the most?

In order to answer these questions, the field of scientometrics and bibliometric offer convenient but debated benchmarking parameters including the impact factor of a scientific journal. Since the journal impact factor is a very superficial measure with no direct relation to the quality of a single scientific work (Carey 2016; Casadevall and Fang 2014) but only providing information about the performance of a specific journal over a relatively short term, other more sophisticated parameters were developed such as the Hirsch (H) index (Hirsch 2005, 2007). They also include a count of the individual citations that a scientific article receives. However, also the H-index is debatable and should not to be used for any purpose (Bertoli-Barsotti and Lando 2017a, b; Bornmann and Leydesdorff 2018). In this respect, experts in the field have coined the expression of amateur bibliometricians describing the uncritical use of bibliometric tools (Bornmann and Leydesdorff 2014).

With these benchmarking options at hand, other questions arise for decision makers: Can we allocate the funding towards a direction that those who did most—get more funding in order to increase their productivity? To which extend can we do so? Is there a ceiling effect? I.e. by which extend is the total (not relative) productivity increased, if we allocate 2 staff positions to a research group which consist of 2 scientists (making a total of 4 scientists then) in comparison to the allocation of 2 staff positions to a group consisting of 20 scientists or of 50 scientists (making it 22 or 52, respectively). Measured in citations? Or in accumulated impact factors or whatsoever? These questions are linked to the so called Matthew effect (Merton 1968): Those who have most get even more. When counting, measuring and benchmarking are done by the use of superficial parameters in an uncritical way by amateur bibliometricians, and (intramural and extramural) funding is allocated on the basis of who performs best in those superficial counts (i.e. total accumulated impact factor count) these questions critically target freedom of research: Scientists or fields who do not produce measurable amounts of superficial parameters such as accumulated journal impact factors will suffer (Lowy 1997).

Further to the question of funding allocation, also career opportunities are critically dependent on bibliometric benchmarking processes and it is common (but critically debated) law: publish or perish (in high impact factor journals) (Jokstad 2016; Publish or perish 2015; Bergquist et al. 2018). Taking these aspects into account, it is obvious that scientometric markers need to be used only with great caution. They should not be easily used to compare scientists of different ages and different fields, institutions or areas with the purpose to cut off funding since research should only be interpreted for quality on the individual level of a published piece of work.

Still, bibliometric parameters can be used to assess gross information contents and evolution of scientific fields over longer periods of time.

It was exactly this purpose when in the years 2005–2009 a new project was started at the Charité in Berlin (Borger et al. 2008; Groneberg-Kloft et al. 2008a, b, 2009a, d, e): A platform termed NewQIS (1.0) was constructed to establish a new approach to visualize research quantity and quality indices (Groneberg-Kloft et al. 2009b, c). NewQIS 1.0 should be used to assess research activities for (1) distinct areas of science, for (2) single institutions, for (3) single countries, or for (4) single time periods (Fig. 1).

Fig. 1.

The NewQIS platform can visualize research parameters for a different areas of science, for b different institutions, for c different countries, or for d different periods of time. A multitude of parameters can be assessed

The platform was intended to be a sound basis for future NewQIS studies in all areas of medicine and science. In the following, we (a) briefly summarize the technical basis, and (b) present an overview of the studies and MD theses which were performed on the basis of NewQIS.

Technical platform

One important aspect of NewQIS 1.0 was to establish a unified technical platform that enables researchers from different fields of science to be able to assess their area of interest. Therefore, a study panel was formed that decided upon the feasibility of the proposed area. The usual applicants were medical students who—in their duty to conduct an MD thesis—submitted search topics to the study panel. After review and affirmation, the NewQIS analysis were performed and raw data was transferred to the applicants for their purpose.

Data acquisition

In NewQIS studies, data is usually retrieved from the Web of Science (WoS) database, i.e. (Kusma et al. 2009). The reason to choose WoS was the ability to perform a citation analysis. This was not possible with PubMed data files.

Depending on the topic of the NewQIS study, the search terms that are entered in the search field consist of various terms which are linked together with Boolean operators such as “AND”, “OR”, “NOT”, i.e. (Glynn et al. 2010).

Depending on the date of the research, the amount of publications and the focus of the research, the evaluation time span covers periods from 1900 until today. Usually, the year in which the NewQIS project is performed, is left out because of incomplete data acquisitions for that given year, i.e. (Al-Mutawakel et al. 2010).

Parameters

The large majority of NewQIS projects focus on a single field of medicine such as a disease and put a focus on the global landscape of research on this particular disease. Thus, the following parameters are usually analyzed (Fig. 1).

Quantity parameters: Productivity

• Total number of published items (i.e. Scutaru et al. 2010b)

• Country specific number of publishes items (i.e. Vitzthum et al. 2010a)

(Semi-)qualitative parameters: Usually, high quality research is characterized by a high number of citations. Therefore, the following citation parameters were also analyzed in the NewQIS projects:

• Total number of citations

• Total number of citations per country

• Country-specific h index

• Country-specific average citation rate per article

Cooperation parameters: A key instrument of NewQIS is to visualize different levels of collaboration. This includes either collaborations between single scientists, countries or institutions. The field of RSV (respiratory syncytial virus) research can give an example how this is achieved: After having identified all relevant RSV-associated publications, the collaborative studies were related to their countries of origin. Publications with two or more authors affiliated to the same country were counted only once for the total number of collaborations of this particular country (Bruggmann et al. 2017c). If an author had two affiliations, these were counted for every country mentioned in the affiliations. Connecting vectors visualized these co-operations; their width and shade of grey reflected the number of joint publications (Bruggmann et al. 2017c). Figure 2 illustrates international collaborations for RSV research.

Fig. 2.

International collaborations for RSV research. International cooperations on RSV research (threshold > 2 cooperations). Numbers in brackets report the number of publications in total/collaborative publications.

https://bmjopen.bmj.com/content/7/7/e013615.long, Data from Bruggmann et al. (2017c)

Visualization

The above listed parameters can also be found in other publications using other approaches (Burak Atci et al. 2019; Ekundayo and Okoh 2018). A specific purpose of NewQIS was to combine these bibliometric parameters with visualization techniques in order to provide a picture of the global landscape of different research aspects. Among different available techniques, density-equalizing map projections (DEMP) were chosen. As elegantly described by Gastner and Newman, map makers searched for a long a way to generate cartograms, in which the sizes of countries appear in proportion to a chosen parameter such as their population (Gastner and Newman 2004). For the purpose of NewQIS, these maps could be used to visualize research activities. As stated by Gastner and Newman, in order to scale countries and still have them properly fit together, they need to be distorted, causing difficulties to read them. In 2004, a new method was proposed which was integrated to the NewQIS platform. With DEMPs being a part of NewQIS, the territories of countries were re-sized according to a particular variable, i.e. in proportion to the countries´ total number of published items regarding to a specific disease. Figure 3 shows examples of DEMPs published within NewQIS studies over the past decade. The distorted global landscape is usually characterized by a dominating USA and an enlarged European area as depicted in Fig. 3a for pulmonary hypertension research output (Gotting et al. 2017). However, there are also research areas in which also countries from other continental regions appear enlarged. This can be seen for snakebite envenoming research for Brazil as shown in Fig. 3b (Groneberg et al. 2016c). China—a rising star in many areas of science—does also appear in some NewQIS assessments prominently as shown for ovarian cancer research in Fig. 3c (Bruggmann et al. 2017d). Concerning Asian countries, a previous assessment of 5527,558 articles has indicated that Asian countries have largely different research focuses in comparison to Western countries (Groneberg-Kloft et al. 2008b). In order to assess changes over the time, spatiotemporal analyses can also be performed by merging to a video consisting of different density-equalizing mapping (Groneberg-Kloft et al. 2009e).

Fig. 3.

Density equalizing map projections (DEMP). a DEMP for pulmonary hypertension research output. Data from Gotting et al. (2017). b DEMP for snakebite envenoming research output. Data from Groneberg et al. (2016c). c DEMP exemplifying prominent Chinese research activities in ovarian cancer research. Data from Bruggmann et al. (2017d)

Topics of NewQIS

Structured MD thesis program

The original concept of NewQIS was a low budget intramural platform which was established without major external funding. In order to be able to assess numerous fields of medicine, medical students were enabled to conduct their MD thesis within the NewQIS platform. The highly structured boundaries of the platform also served as a quality control for the results of the thesis projects making scientific misconduct very difficult (since there was no possibility for the students to manipulate the algorithms applied by the platform).

Since 2009, nearly 80 theses were completed using the methodology of the platform making NewQIS one of the most successful structured thesis programs in Germany. As tutors/mentors of the theses, seven associate/full professors served so far. Also, two technical tutors were present to oversee calculations and data management. Table 1 lists the medical thesis topics.

Table 1.

Medical thesis projects that applied the structured NewQIS program

No.	Place of thesis	Year of exam	Topic of MD thesis	Analysis interval	References
1.	Frankfurt	2019	Tunisia	1900–2013	Fuchs (2019)
2.	Frankfurt	2018	Immigration	1900–2016	Trost (2018)
3.	Frankfurt	2018	Cervical cancer	1900–2015	Quinkert (2018)
4.	Frankfurt	2018	Ovarian carcinoma	1900–2014	Pulch (2018)
5.	Frankfurt	2018	Noise	1900–2014	Brich (2017)
6.	Frankfurt	2018	Needle stick injury	1900–2014	Braumann (2017)
7.	Frankfurt	2018	Jaw palate clefts	1900–2014	Mierke (2018)
8.	Frankfurt	2018	Child abuse	1900–2014	Wolf (2018)
9.	Frankfurt	2018	Melanoma	1900–2014	Scholz (2018)
10.	Frankfurt	2018	Rotavirus	1900–2013	Köster (2018)
11.	Frankfurt	2018	Caesarean section	1900–2013	Löhlein (2018)
12.	Frankfurt	2018	Caries	1900–2012	Kröber (2018)
13.	Frankfurt	2018	Tuberculosis	1900–2012	Weber (2018)
14.	Frankfurt	2018	Tonsillectomy	1900–2014	Neuenfeldt (2018)
15.	Frankfurt	2018	Schizophrenia	1900–2015	Lammer (2018)
16.	Frankfurt	2018	Gestational diabetes	1900–2012	Richter (2018)
17.	Frankfurt	2017	Pancreatic carcinoma	1900–2013	Krempel (2017)
18.	Frankfurt	2017	Osteoporosis	1900–2012	Mäule (2017)
19.	Frankfurt	2017	Human papilloma virus	1900–2009	Kayser (2017)
20.	Frankfurt	2017	Cholera	1900–2009	Mühlbach (2017)
21.	Frankfurt	2016	Psychiatric journals	1920–2012	Abberger (2016)
22.	Frankfurt	2016	Gastroenterological journals	1945–2012	Schäfer (2016)
23.	Frankfurt	2016	Public Health	1912–2012	D. Hoffmann (2016)
24.	Frankfurt	2016	Toxoplasmosis	1900–2012	Handl (2016)
25.	Frankfurt	2015	Aortic aneurysm	1900–2010	Ofosu (2015)
26.	Berlin	2015	Depression, suicide, cannabis, bipolar disorder	1900–2008	Vogelzang (2015)
27.	Frankfurt	2014	Yellow fever	1900–2012	Bundschuh (2013)
28.	Frankfurt	2014	Smoking and pregnancy	1900–2005	Mund (2013)
29.	Frankfurt	2014	Propofol	1977–2009	W. Weiland (2014)
30.	Frankfurt	2014	Extrinsic allergic alveolitis	1900–2007	Walger (2014)
31.	Frankfurt	2013	Hepatitis B	1900–2010	Schmidt (2013)
32.	Frankfurt	2013	Osteomyelitis	1900–2009	Schwartzmann (2013)
33.	Frankfurt	2013	Passive smoking	1900–2009	Jacobus (2013)
34.	Frankfurt	2013	Poliomyelitis	1900–2009	Drews (2012)
35.	Frankfurt	2013	Dental implants	1900–2010	Albrecht (2013)
36.	Frankfurt	2013	Influenza	1900–2009	Fricke (2011)
37.	Berlin	2012	Diabetic retinopathy	1900–2008	Wahrlich (2012)
38.	Berlin	2012	Neurologic and psychiatric rehabilitation	1900–2009	Hoffmann–Roe (2012)
39.	Frankfurt	2012	MRI-Scan	1981–2006	Schwarze (2012)
40.	Berlin	2012	Pulmonary hypertension	1900–2007	Götting (2012)
41.	Frankfurt	2012	Allergic rhinitis	1900–2007	Wende (2012)
42.	Frankfurt	2012	Sarcoidosis	1900–2008	Kirchdörfer (2012)
43.	Berlin	2011	Bacterial meningitis	1900–2007	Pleger (2011)
44.	Berlin	2011	Erythropoietin	1900–2007	Schöffel (2011)
45.	Berlin	2011	Bladder cancer	1909–2007	Domnitz (2011)
46.	Berlin	2011	Borrelia burgdorferi	1900–2008	Scholz (2011)
47.	Frankfurt	2011	Dengue virus infections	1900–2007	Müller (2011)
48.	Hannover	2011	Orthopedic diseases	1900–2008	Vitzthum (2011)
49.	Berlin	2011	Glioblastoma multiforme	1900–2008	Addicks (2011a)
50.	Berlin	2011	Infectious endocarditis	1900–2008	Berkholz (2011)
51.	Berlin	2011	Obesity	1900–2009	Franke (2011)
52.	Berlin	2011	Air pollution, particulate matter and sulphur dioxide	1955–2006	Zell (2011)
53.	Berlin	2011	Methicillin-resistant staphylococcus aureus (MRSA)	1961–2007	Addicks (2011b)
54.	Berlin	2011	Barotrauma	1900–2008	Garnew (2011)
55.	Berlin	2011	M Alzheimer	1985–2006	Tropp (2011)
56.	Berlin	2011	Varicella zoster virus	1900–2008	Busch (2011)
57.	Berlin	2011	Resuscitation	1900–2007	Weiland (2011)
58.	Berlin	2011	Cystic fibrosis	1900–2009	Falahkohan (2011)
59.	Berlin	2010	Body mass index	1900–2008	Bohlen (2010)
60.	Berlin	2010	Poisonous snake bites	1900–2007	Geier (2010)
61.	Berlin	2010	Myasthenia gravis	1900–2008	Koch (2010)
62.	Berlin	2010	Asbestos	1900–2008	Kröger (2010)
63.	Berlin	2010	Clostridium botulinum	1905–2008	Uibel (2010)
64.	Berlin	2010	Age-related macular degeneration	1900–2008	Steinberg (2010)
65.	Berlin	2010	SARS	2003–2007	Kreiter (2010)
66.	Berlin	2010	Epithelial precursor lesions	1900–2008	Grajewski (2010)
67.	Berlin	2010	Multiple sclerosis	1900–2008	Hoffmann (2010)
68.	Berlin	2010	Herpes simplex virus	1900–2007	Szerwinski (2010)
69.	Berlin	2009	Burnout syndrome	1983–2006	Fröhlich (2009)
70.	Berlin	2009	Drowning accidents	1900–2006	Schilling (2010)
71.	Berlin	2009	Measles	1900–2008	Rospino (2009)
72.	Berlin	2009	Human immunodeficiency virus (HIV)	1982–2007	Neye (2009)
73.	Berlin	2009	Carpal tunnel syndrome	1900–2006	Friedebold (2009)
74.	Berlin	2009	Streptococcus	1957–2006	Bock (2009)
75.	Berlin	2009	Syphilis	1900–2007	Bircks (2010)
76.	Berlin	2009	Arthrosis	1900–2007	Mayer (2009)
77.	Berlin	2009	Telemedicine	1976–2006	Rahimian (2009)
78.	Berlin	2009	Epilepsy	1900–2007	Bircks (2010)
79.	Berlin	2009	Bronchial asthma	1967–2006	Puk (2009)

Scientific publications

Since 2008, more than 80 studies using the NewQIS platform were published after peer review. The majority of them based on medical thesis projects with the MD students being first authors in case of writing the manuscripts or co-authors of the scientific studies. The topics ranged from infectious diseases, infectious agents, to cancers, neurological or psychiatric disorders, lung diseases or other diseases. Apart from diseases, they also encompassed i.e. public health issues including tobacco control, medical procedures or techniques. In total, more than 1.6 million published articles related to specific search terms were analyzed for the above listed parameters. Table 2 provides an overview of the different NewQIS articles.

Table 2.

Scientific publications applying the NewQIS platform

No.	Authors	Year	Title	Analysis interval	Total number of published items	Reference
1.	Börger et al.	2008	Models of asthma: density-equalizing mapping and output benchmarking	1900–2006	3489	Borger et al. (2008)
2.	Groneberg-Kloft et al.	2008	Institutional operating figures in basic and applied sciences: scientometric analysis of quantitative output benchmarking	1966–1976 1996–2006	5,527,558	Groneberg-Kloft et al. (2008b)
3.	Groneberg-Kloft et al.	2009	Cough as a symptom and a disease entity: scientometric analysis and density-equalizing calculations	1900–2007	12,960	Groneberg-Kloft et al. (2009a)
4.	Groneberg-Kloft et al.	2009	Inter-disease comparison of research quantity and quality: bronchial asthma and chronic obstructive pulmonary disease	1987–2006	n.a.	Groneberg-Kloft et al. (2009d)
5.	Kusma et al.	2009	Tobacco control: visualisation of research activity using density-equalizing mapping and scientometric benchmarking procedures	1952–2008	1846	Kusma et al. (2009)
6.	Schöffel et al.	2009	The role of endocarditis, myocarditis and pericarditis in qualitative and quantitative data analysis	1900–2007	18,967 (endocarditis) 7803 (myocarditis) 5552 (pericarditis)	Schoffel et al. (2009)
7.	Vitzthum et al.	2009	Scoliosis: density-equalizing mapping and scientometric analysis	1904–207	8186	Vitzthum et al. (2009)
8.	Al-Mutawakel et al.	2010	Scientometric analysis of the world-wide research efforts concerning Leishmaniasis	1957–2006	19,277	Al-Mutawakel et al. (2010)
9.	Bohlen et al.	2010	Scientometric analysis of the BMI	1900–2008	63,845	Bohlen et al. (2010)
10.	Glynn et al.	2010	Breast cancer research output, 1945–2008: a bibliometric and density-equalizing analysis	1945–2008	180,126	Glynn et al. (2010)
11.	Grajewski et al.	2010	A scientometric analysis of leukoplakia and erythroplakia	1900–2008	2659	Grajewski et al. (2010)
12.	Schöffel et al.	2010	Reumatoid arthritis: scientific development from a critical point of view	1901–2007	78,128	Schoffel et al. (2010a)
13.	Scutaru et al.	2010	Density-equalizing mapping and scientometric benchmarking of European allergy research	2001–2007	n.a.	Scutaru et al. (2010a)
14.	Schöffel et al.	2010	Arthroplasty: critical scientometric analysis of current benchmarking and evaluation procedures	1901–2007	21,874	Schoffel et al. (2010b)
15.	Scutaru et al.	2010	Density-equalizing mapping and scientometric benchmarking in Industrial Health	1900–2014	n.a.	Scutaru et al. (2010b)
16.	Schöffel et al.	2010	Critical analysis of publication procedures and evaluation regarding ankylosing spondylitis by density-equalizing mapping and scientometric methods	01–2007	8156	Schoffel et al. (2010c)
17.	Vitzthum et al.	2010	Scientometric analysis and combined density-equalizing mapping of environmental tobacco smoke (ETS) research	1900–2008	6580	Vitzthum et al. (2010a)
18.	Zell et al.	2010	Air pollution research: visualization of research activity using density-equalizing mapping and scientometric benchmarking procedures	1955–2006	26,253	Zell et al. (2010)
19.	Vitzthum et al.	2010	Cardiac insufficiency: a critical analysis of the current publication procedures under quantitative and qualitative aspects	1900–2007	82,828	Vitzthum et al. (2010)
20.	Mache et al.	2010	Alzheimer’s Disease—a Scientometric Analysis and Data Acquisition	1985–2008	50,030	Mache et al. (2010)
21.	Groneberg et al.	2011	Drowning-a scientometric analysis and data acquisition of a constant global problem employing density equalizing mapping and scientometric benchmarking procedures	1900–2006	2381	Groneberg et al. (2011)
22.	Healy et al.	2011	The h index and the identification of global benchmarks for breast cancer research output	1945–2008	n.a.	Healy et al. (2011)
23.	Van Mark et al.	2011	Shift- and Nightwork—a scientometric analysis	1900–2008	3092	van Mark et al. (2011)
24.	Vogelzang et al.	2011	Depression and suicide publication analysis, using density equalizing mapping and output benchmarking	1900–2007	6069	Vogelzang et al. (2011)
25.	Glynn et al.	2012	Laryngeal cancer: quantitative and qualitative assessment of research output, 1945–2010	1945–2010	8658	Glynn et al. (2012)
26.	Vogelzang et al.	2012	A bibliometric analysis of bipolar affective disorders using density-equalizing mapping and output benchmarking	1900–2008	18,831	Vogelzang et al. (2012)
27.	Bundschuh et al.	2013	Yellow fever disease: density equalizing mapping and gender analysis of international research output	1900–2012	5053	Bundschuh et al. (2013)
28.	Fricke et al.	2013	Influenza: a scientometric and density-equalizing analysis	1900–2009	51,418	Fricke et al. (2013)
29.	Gerber et al.	2013	Gout: a critical analysis of scientific development	1990–2012	4424	Gerber et al. (2013)
30.	Groneberg-Kloft et al.	2013	Traffic medicine-related research: a scientometric analysis	1900–2008	5193	Groneberg-Kloft et al. (2013)
31.	Schmidt et al.	2014	Hepatitis B: global scientific development from a critical point of view	1971–2011	49,166	Schmidt et al. (2014)
32.	Addicks et al.	2014	MRSA: a density-equalizing mapping analysis of the global research architecture	1961–2007	7671	Addicks et al. (2014)
33.	Carl et al.	2014	Curare-a curative poison: a scientometric analysis	1900–2013	3867	Carl et al. (2014)
34.	Gerber et al.	2014	Antineutrophil cytoplasmic antibody-associated vasculitides: a scientometric approach visualizing worldwide research activity	1993–2013	6216	Gerber et al. (2014a)
35.	Mund et al.	2014	Global research on smoking and pregnancy-a scientometric and gender analysis	1900-2012	10,043	Mund et al. (2014)
36.	Gerber et al.	2014	A scientometric analysis of global research activity during the last 35 years	1972–2012	11,839	Gerber et al. (2014b)
37.	Gerber et al.	2014	Silicosis: geographic changes in research: an analysis employing density-equalizing mapping	1920–2012	2805	Gerber et al. (2014c)
38.	Pleger et al.	2014	Bacterial meningitis: a density-equalizing mapping analysis of the global research architecture	1900–2007	7998	Pleger et al. (2014)
39.	Brüggmann et al.	2015	Congenital toxoplasmosis: an in-depth density-equalizing mapping analysis to explore its global research architecture	1900–2012	13,044	Bruggmann et al. (2015a)
40.	Geaney et al.	2015	Type 2 Diabetes Research Yield, 1951-2012: Bibliometrics Analysis and Density-Equalizing Mapping	1951–2012	24,783	Geaney et al. (2015)
41.	Brüggmann et al.	2015	Caesarean Section-A Density-Equalizing Mapping Study to Depict Its Global Research Architecture	1900–2013	12,608	Bruggmann et al. (2015b)
42.	Groneberg et al.	2015	Density equalizing mapping of obesity: analysis of a global epidemic	1900–2009	94,987	Groneberg et al. (2015a)
43.	Ohlendorf et al.	2015	Arthrosis: a scientometric analysis	1900–2013	46,212	Ohlendorf et al. (2015a)
44.	Groneberg et al.	2015	Telemedicine—a scientometric and density equalizing analysis	1900–2006	3290	Groneberg et al. (2015b)
45.	Quarcoo et al.	2015	Ebola and Its Global Research Architecture-Need for an Improvement	1976–2014	3081	Quarcoo et al. (2015)
46.	Groneberg et al.	2015	Density equalizing mapping of the global tuberculosis research architecture	1900–2012	58,319	Groneberg et al. (2015c)
47.	Ohlendorf et al.	2015	Magnetic resonance imaging Density equalizing mapping analysis of global research architecture	1981–2007	49,122	Ohlendorf et al. (2015b)
48.	Brüggmann et al.	2016	Endometriosis and its global research architecture: an in-depth density-equalizing mapping analysis	1900–2009	11,056	Bruggmann et al. (2016a)
49.	Groneberg et al.	2016	Pancreatitis: Global Research Activities and Gender Imbalances: A Scientometric Approach Using Density-Equalizing Mapping	1900–2012	27,826	Groneberg et al. (2016a)
50.	Brüggmann et al.	2016	World-wide architecture of osteoporosis research: density-equalizing mapping studies and gender analysis	1900–2012	57,453	Bruggmann et al. (2016b)
51.	Köster et al.	2016	Rotavirus—Global research density equalizing mapping and gender analysis	1900–2013	5906	Koster et al. (2016)
52.	Brüggmann et al.	2016	Global architecture of gestational diabetes research: density-equalizing mapping studies and gender analysis	1900–2012	12,504	Bruggmann et al. (2016c)
53.	Groneberg et al.	2016	Snakebite Envenoming—A Combined Density Equalizing Mapping and Scientometric Analysis of the Publication History	1900–2016	17,998	Groneberg et al. (2016c)
54.	Schöffel et al.	2016	Ulcerative colitis: A scientometric approach to the global research output and network	1900–2016	40,343	Schoffel et al. (2016a)
55.	Schreiber et al.	2016	Patient safety: the landscape of the global research output and gender distribution	1963–2014	4079	Schreiber et al. (2016)
56.	Schöffel et al.	2016	A critical perspective on the global research activity in the field of bladder cancer	1900–2007	19,651	Schoffel et al. (2016b)
57.	Groneberg et al.	2016	Analysis of the research architecture on the burnout syndrome	1983–2006	3146	Groneberg et al. (2016b)
58.	Schöffel et al.	2016	Sarcoidosis: A Descriptive Approach to the Global Research Network and Recent Scientific Developments	1900–2008	14,190	Schoffel et al. (2016c)
59.	Groneberg-Kloft et al.	2016	Analysis of research architecture in the field of psychiatric rehabilitation	1900–2009	9271	Groneberg-Kloft et al. (2016)
60.	Schöffel et al.	2016	Pancreatic Cancer-Critical Examination of the Global Research Architecture and Recent Scientific Developments	1900–2013	11,445	Schoffel et al. (2016d)
61.	Brüggmann et al.	2017	Polycystic ovary syndrome: analysis of the global research architecture using density equalizing mapping	1900–2014	6261	Bruggmann et al. (2017a)
62.	Groneberg et al.	2017	Glioblastoma research: US and international networking achievements	1900–2008	14,411	Groneberg et al. (2017)
63.	Brüggmann et al.	2017	Ectopic pregnancy: exploration of its global research architecture using density-equalizing mapping and socioeconomic benchmarks	1900–2012	8040	Bruggmann et al. (2017b)
64.	Götting et al.	2017	Pulmonary Hypertension: Scientometric Analysis and Density-Equalizing Mapping	1900–2007	18,986	Gotting et al. (2017)
65.	Brüggmann et al.	2017	Respiratory syncytial virus: a systematic scientometric analysis of the global publication output and the gender distribution of publishing authors	1900–2013	4600	Bruggmann et al. (2017c)
66.	Schöffel et al.	2017	Hirschsprung Disease: Critical Evaluation of the Global Research Architecture Employing Scientometrics and Density-Equalizing Mapping	1900–2015	2978	Schoffel et al. (2017a)
67.	Brüggmann et al.	2017	Ovarian cancer: density equalizing mapping of the global research architecture	1900-2014	23,378	Bruggmann et al. (2017d)
68.	Schöffel et al.	2017	Evaluation of the Global Research Architecture Regarding Diabetic Retinopathy	1900–2008	15,624	Schoffel et al. (2017b)
69.	Brüggmann et al.	2017	Maternal depression research: socioeconomic analysis and density-equalizing mapping of the global research architecture	1900–2012	7330	Bruggmann et al. (2017e)
70.	Groneberg	2018	Biomedical Research in Wroclaw: A Combined Density-Equalizing Mapping and Scientometric Analysis	1972–2016	10,366	Groneberg (2018a)
71.	Brüggmann et al.	2018	World-wide research architecture of vitamin D research: density-equalizing mapping studies and socio-economic analysis	1900–2014	25,992	Bruggmann et al. (2018a)
72.	Groneberg et al.	2018	The story behind Oncotarget? A bibliometric analysis	2010–2017	21,961	Groneberg et al. (2018)
73.	Brüggmann et al.	2018	Human papilloma virus: global research architecture assessed by density-equalizing mapping	1900–2009	29,330	Bruggmann et al. (2018b)
74.	Klingelhöfer et al.	2018	Fifteen years after September 11: Where is the medical research heading? A scientometric analysis	2001–2016	4250	Klingelhofer et al. (2018a)
75.	Brüggmann et al.	2018	The uterine fibroid/myoma tumour: analysis of the global research architecture using density-equalizing mapping	1900–2015	6176	Bruggmann et al. (2018c)
76.	Groneberg	2018	Social sciences research in the Central European city of Wroclaw: A density-equalizing mapping analysis	1966–2017	1787	Groneberg (2018b)
77.	Lammer et al.	2018	Development of the global schizophrenia research under epidemiological and socio-economic influences	1900–2015	42,492	Lammer et al. (2018)
78.	Trost et al.	2018	Immigration: analysis, trends and outlook on the global research activity	1900–2016	6763	Trost et al. (2018)
79.	Klingelhöfer et al.	2018	Aflatoxin—Publication analysis of a global health threat	1900–2006	5122	Klingelhofer et al. (2018b)
80.	Schöffel et al.	2018	Crohn’s Disease: A Critical Approach to Publication Procedures and Citation Behavior of the Global Research Network	1900–2013	45,259	Schoffel et al. (2018)
81.	Trost et al.	2018	Immigration: analysis, trends and outlook on the global research activity	1900–2016	6763	Trost et al. (2018)
82.	Groneberg	2019	Academic chemistry and related fields in Wroclaw: Density-equalizing mapping studies over the past decades	1972–2016	15,267	Groneberg (2019)

Limitations of NewQIS

There are numerous limitations present in every NewQIS-based study:

1. As with every other bibliometric approach, also NewQIS is limited to the data base it uses. Although producing global landscapes of research, it should never be forgotten that these pictures only delineate the research output which can be found in a specific data base (i.e. Web of Science) with a specific search term. Thus, all research not listed in the WoS and all research excluded by the search term (no search term can be absolutely perfect) is not included in the global landscape. This needs to be taken into account carefully when NewQIS results are interpreted. Especially the language bias constitutes an important problem: journals published in English have a higher chance of getting included to the data bases (Nieminen and Isohanni 1999). Thus, non-English speaking countries are underrepresented concerning their research activities and important but regional data such as regional epidemiologic data is not identified (Pleger et al. 2014).

2. A further limitation that needs to be addressed is the above Matthew effect mentioned above: Communication systems in science are directed towards a reward of highly productive and renowned scientists and institutions. This leads to a pyramidal citation scheme (Merton 1968; Pleger et al. 2014).

3. The so-called (semi-)qualitative indicators that are used in NewQIS are parameters such as the total citations, citation rate, country-specific h-index. They need to be interpreted very carefully. As already earlier critically discussed, they are not real measures for the quality of individual research (Pleger et al. 2014). In this respect, a recent study addressed the question if methodological quality and completeness of reporting are associated with citation-based measures of publication impact (Mackinnon et al. 2018). The authors performed a secondary analysis of a systematic review of dementia biomarker studies. They reported that citation rates and 5-year journal impact factors appear to measure different dimensions. While citation rates were weakly associated with completeness of reporting, none of these metrics was related to methodological rigor. They suggested that high publication usage and journal outlet is not a guarantee of quality and readers should critically appraise all papers regardless of presumed impact (Mackinnon et al. 2018). Therefore, qualitative aspects are better addressed by advanced meta-analysis approaches using i.e. Cochrane systematics (Stovold et al. 2014).

Further issues

Scientometrics as research area is a niche within science. Funding is difficult to acquire for scientometric projects. However, it is the long term aim of NewQIS to analyze about 200 different areas within the next decade and to repeat assessments in 5- to 10-year intervals of important areas in order to assess changes in global research activities. When counting the raw data analyzed in the first 100 projects, we approximately invested about 50,000 work hours. Without extramural funding, this was only achievable by the workforce of medical students who performed their MD projects within NewQIS. In contrast to peer reviewed scientific reports which have been published for different NewQIS studies, a German medical thesis usually encompasses a much longer manuscript with 80–100 pages. This has been achieved by the medical students by writing comprehensive introductions about the field of research they analyze within their thesis. Thereby, they demonstrate that they possess an extensive knowledge about their thesis project. This is a prerequisite to obtain an MD degree. Also, the thesis students have to write detailed descriptions of their methodological approach (the NewQIS techniques) in the methods sections of the thesis and they have to discuss limits of the methodology in the discussion sections of their thesis.

This leads to two potential pitfalls:

1. In the case of the methods sections, the thesis students have to follow strictly the above described protocols of NewQIS. This technical overlap is important and a strength of the platform in order to facilitate the comparison of results between the different diseases studied. However, it can be anticipated, that the use of these stringent protocols in nearly 80 different thesis projects—all with different target areas, i.e. ranging from burnout syndrome (Fröhlich 2009) to bronchial asthma (Puk 2009)—brings the same problem as rewriting a passage on the methodology of other highly structured techniques such as RT-PCR (reverse transcriptase-polymerase chain reaction) which has now been published more than 250,000 times according to the PubMed. As with nearly identical descriptions of PCR and other molecular biology methods which can be found in peer reviewed scientific papers, an overlapping wording does not represent an act of plagiarism but rather exemplifies the impossibility to reword a similar methods section for more than 80 times without overlapping sentences. This does also apply for the part of the discussion in which the methodology and its limitations are discussed. Addressing these issues, the international Committee on Publication Ethics (COPE) points to a guideline of BioMed Central editors which outlines the following: “Use of similar or identical phrases in methods sections where there are limited ways to describe a common method, (…), is not uncommon. In such cases, an element of text recycling is likely to be unavoidable in further publications using the same method. Editors should use their discretion when deciding how much overlap of methods text is acceptable, considering factors such as whether authors have been transparent and stated that the methods have already been described in detail elsewhere and provided a citation” (COPE) (https://publicationethics.org/text-recycling-guidelines). Therefore, to overcome this pitfall, peer-reviewed NewQIS studies cite previous studies because of the methodological similarities—which are a strength of the platform. Also, thesis students are urged to cite every other NewQIS thesis which used the platform and to declare that the used methodology is part of NewQIS and therefore similar (apart from i.e. the different search terms).

2. The introductions of the respective thesis usually follow the guidelines of up-to-date reviews i.e. on the disease which is analyzed for the thesis. In this respect, numerous introductions from NewQIS related thesis projects were also published as CME (continuing medical education) articles or as narrative reviews. Unfortunately, a recent analysis showed that within one thesis project, almost all parts of the introduction were copied by the student from the Wikipedia—a case of severe plagiarism that led to the deprivation of the Dr. med. degree (MD Thesis) of the student (Sudik 2011). In order to prevent future cases of plagiarism, all medical thesis now need to be analyzed within a plagiarism check prior to the official submission of the thesis to the medical school.

Future of NewQIS

The NewQIS platform will be used as NewQIS 2.0 in a next decade of further scientometric studies. There will be the following issues:

Project of 200 As stated earlier, NewQIS 2.0 is intended to encompass about 200 different search projects with all areas of medicine, life sciences and also other areas of science in the next 10 years. Also, projects carried out 10 years ago and reported worldwide research activities (in the Web of Science) until 2005, should now be repeated in order to investigate the development of scientific activities.

New focuses Originally conceived as a tool to investigate publication activities in single areas of medicine, i.e. in different infectious diseases, NewQIS has also proven to be a valuable tool for other purposes, i.e. to analyze journals (Scutaru et al. 2010b; Groneberg et al. 2018). Also, it could be used for the analysis of cities with regard to research activities of affiliations in these cities. A recent example was the so-called NewQIS-Wroclaw project that assessed scientific activities in the Central European Polish city in three different areas: biomedical research, chemical research and social sciences (Groneberg 2018a, b, 2019) and demonstrated a strong increase over the past decades.

New parameters As introduced in the past years, NewQIS studies may also focus upon socio-economic features. In this respect, various economic key figures were used. I.e. two quotients were calculated to assess the scientific output of a specific country for RSV research (Bruggmann et al. 2017c):

1. in relation to the number of inhabitants (Q1)

2. in relation to its economic power (as measured by the gross domestic product, GDP, Q2) (Bruggmann et al. 2017c). Data regarding the population and GDP of investigated countries was obtained from 2012 The quotients were calculated as follows:

   1. Articles/population index (Q1) = number of articles/population in million inhabitants
   2. Articles/GDP index (Q2) = number of articles/GDP in 1000 billion US-Dollars

Within the RSV research NewQIS study, also, all countries were classified into high-income, upper-middle-income, lower-middle-income and low-income groups according to World Bank definitions (Bruggmann et al. 2017c). Then, the total number of RSV articles was related to the gross domestic expenditure on Research and Development (R&D in % of GDP) as well as to the number of researchers (per million inhabitants) affiliated to the investigated countries.

Conclusion

For over 10 years, the NewQIS platform has been used as a tool for peer reviewed scientific studies and for medical thesis in order to study numerous fields of science. As NewQIS 2.0 the project now heads into the next decade with a variety of new aspects in focus such as detailed socio-economic analysis or gender aspects. Using density equalizing mapping projections thousands of new pictures of global research landscapes will be generated. With numerous novel aspects that have been introduced to NewQIS within the past years, the platform will be a helpful tool for different aspects of scientometrics in the future.

Authors’ contributions

DAG, DK, DB, CS, AF and DQ conceived the review, and participated in the process of drafting the manuscript. All authors read and approved the final manuscript.

Compliance with ethical standards

Conflict of interests

The authors declare that they have no competing interests.