Assessing Clinical and Life Sciences Performance of Research Institutions in Split, Croatia, 2000-2006

Abstract

Aim

To evaluate publications of clinical and life scientists from research institutions in Split, Croatia, and the publication output from government-funded research projects of the University of Split School of Medicine.

Methods

We analyzed the number of publications from research institutions in Split, Croatia, in the 2000-2006 period, relative impact factors, predominant research fields, output of researchers from the University of Split School of Medicine receiving government research grants, and the average price of published article.

Results

From 2000 to 2006, clinical and life scientists published 350 articles indexed in Thomson Scientific database Current Contents. The number of articles increased from 30 in 2000 to 76 in 2006, and the average impact factor of journals where these articles were published increased from 2.03 in 2000 to 2.89 in 2006. Twenty percent of articles (72/350) were published in the Croatian Medical Journal. Principal investigators of the 12 research projects receiving government grants published 0 to 8 articles related to the project topic in the 2002-2006 research grant cycle. The research grantees published 78 original research articles, with an average price per article of € 29.210.

Conclusion

Although the number and impact factor of research articles published by clinical and life scientists from Split, Croatia, is increasing, it is still low when the number of scientists is taken into account. There should be better mechanisms of control and evaluation of research performance of government-funded research projects.

Science in Croatia is funded mostly by the government, ie, the Ministry of Science, Education, and Sports (MSES). In 2003, the government budget for science in Croatia was HRK 684 million (€ 93 million), increasing to HRK 813 million (€ 110 million) in 2006 (1). The gross domestic product (GDP) of Croatia in 2003 was HRK 198 422 million (€ 27 934 million), increasing to HRK 250 590 million (€ 34 015 million) in 2006 (2). Thus, the budget for science was 0.34% of GDP in 2003 and 0.32% in 2006. However, the Croatian government claims that science, technology, and education are the key factors in the process of integration to the European Union and transition from an industrial to knowledge economy (3,4).

After a large part of public funds have been invested into science, it is reasonable to ask if the current amount of funding in Croatia has a measurable impact on the number and quality of research articles. Current evaluation practices of research performance of government-funded research grants in Croatia are almost non-existing. Croatian scientists whose research is funded by the government should be able to provide a sustained track record of significant output in peer-reviewed literature and show a strong commitment to increased quality research output.

Data on the cost of research in Croatia are scarce. Some attention has recently been devoted to the number of the Thomson Scientific Science Citation Index (SCI) articles produced by Croatian scientists (5-7), but there are no publications assessing the results of government-funded research projects or calculating the costs of publication in Croatia.

The aim of this study was to assess research productivity of clinical and life scientists from research institutions in Split, Croatia, as a representative example of a small academic community in Croatia. The second aim was to compare the research output with financial input. The third aim was to discuss current evaluation practices of scientific productivity in Croatia and most appropriate scientometric indicators for assessing research performance of Croatian clinical and life scientists.

Methods

The study covered a period of 6 years, from 2000 to 2006, a period long enough to estimate research output of the last 2002-2006 MSES grant cycle. The analysis included data on the number of publications, number of researchers, number of co-authors, the extent of collaborations, Journal Citation Reports (JCR) impact factor and relative impact factor, and financial investment through grants and capital equipment.

Setting

The major research institution in Split, Croatia, is the University of Split, a public university founded in 1974. It is the second largest university in Croatia, with around 18 000 students. As the predominant scientific institution in South Croatia, the University of Split has expanded over the past years and now includes 11 schools, 3 university departments, and several other institutes and scientific departments.

In 2003, the Mediterranean Institute for Life Sciences (MedILS) was founded in Split as a non-profit organization and officially opened for research in 2007 (8).

The MSES research project funding is the basic system of funding public research activities in Croatia. All financial control mechanisms for government-funded research and development in Croatia are applied by the MSES; it is the Ministry that allocates funds for material expenditure, research projects, junior researcher employment, and approval of vacancies for new appointments. Research institutions are funded from two main budget sources: a) funds for research or academic institutions – basic salaries, basic overheads, operational costs, equipment, and capital outlays, and b) research grants obtained through a major competitive process (9).

Only Croatian scientists with a doctoral degree (PhD and MD, PhD) are eligible to apply to research grants. Salaries of research staff, including principal investigators, laboratory technicians, and junior researchers, are also paid by the government. Beside research grants and salaries, the government also provides grants for research equipment. Universities and Schools sometimes also allocate a part of their budget to research purposes.

Scientometric indicators

As a measure of research output of Split researchers, the number of publications indexed in two Current Contents databases, Life Sciences and Clinical Medicine (Institute for Scientific Information – Thomson Scientific, Philadelphia, PA, USA), was assessed for the 2000-2006 period. This database was used because it provides information on impact factors.

Due to a large inconsistency in the addresses of research institutions, we opted for the name of the city (Split) as the search word for the retrieval of data. Thus, the Current Contents database search was performed through OVID Web Gateway using the search syntax “Split.in.” limited to subsets and a particular year (example of a full syntax: 1) Split.in. 2) limit 1 to yr = ”2004” 3) limit 2 to sb = ”life”).

We retrieved a total of 354 articles published in the Clinical Medicine and Life Sciences section by authors with “Split” in the author address. Only original research were included in the study. Some articles were indexed in both Current Contents categories and duplicates were excluded. All author addresses of retrieved articles were additionally checked in case the word “split” appeared in the name of a research institution that is not from Split, Croatia. Three such cases were found as follows: an article published by an author affiliated with the New England Surgical Center Retreat Split Rock and two authors reporting on Studies of Pediatric Liver Transplantation (SPLIT). One additional article was deemed unsuitable because it was an abstract from a scientific meeting rather than a full-length journal article. These four publications were removed from the database, leaving 350 articles in the data set.

The addresses of all the authors were assessed to estimate an extent of collaboration between Croatian and international researchers. The authors' addresses were also used to determine most productive Split-based institutions in the Clinical and Life Sciences section.

Thomson Scientific JCR was used to obtain annual impact factors for journals for the year when a manuscript was published (10).

The comparison of impact factors between different fields can be misleading and, therefore, we calculated the relative impact factors as a percentile in the JCR Subject Category, showing a relative standing of a journal among all other journals within the same research field (JCR Subject Category). The percentile was calculated for all JCR Subject Categories in which journals were categorized. For example, journal Clinical Infectious Diseases is listed in the following three JCR Subject Categories: Infectious diseases, Immunology, and Microbiology. When we calculated the percentiles of the journal in all JCR subject categories in which a journal is indexed, the best one was selected for our evaluation purposes.

Archival data

The number of scientists at the University of Split School of Medicine was obtained from the School's archives, as this is the largest research institution in Split, with readily available data. As all scientists in the School have a teaching duty, we analyzed semi-annual teaching reports, in which all teachers are listed with their titles and affiliations.

Article subject and grant subject

To assess the productivity of research grantees from the University of Split School of Medicine, we compared the title and description of their research projects with the titles and abstracts of the articles they published during the 2002-2006 grant cycle. All four authors performed this analysis and reached a consensus as to which articles had been published on the research project topic and which had been published in the same research field but were not specifically related to the topic of the research project.

Our research grant analysis revealed that the majority of principal investigators (11 out of 12) published articles co-authored by other research grantees, so that one article was attributed to two or more grants. Therefore, the percentage of overlap was calculated by dividing the number of articles co-authored by other grantees with the number of all articles published by a principal investigator.

Articles-per-scientist ratio and the cost of publications

To assess the costs of publication, we analyzed only the publications from School of Medicine because most publications from Split in the 2000-2006 period were authored by clinical and life scientists affiliated to the School (89%).

In the study period, there was only one full MSES grant cycle (2002-2006) at the University of Split School of Medicine, with 12 principal investigators obtaining MSES grants. Number of grants and their budget were used as input measures. Output measures were articles published in journals cited in Current Contents, impact factor of those journals, and their relative impact factor.

Results

Scientometric indicators for clinical and life sciences at research institutions in Split, Croatia (2000-2006)

From 2000 to 2006, the authors from research institutions in Split published 350 articles indexed in the Clinical Medicine and Life Sciences subsets of the Current Contents. The number of publications in these categories tripled in the observed period, from 30 publications in 2000 to 76 publications in 2006 (Figure 1). Most publications were published by authors from the University of Split School of Medicine, with a very small contribution from other research institutions in Split (Table 1).

Figure 1.

Publications of clinical and life scientists from research institutions in Split, Croatia, 2000-2006. Open bars – life sciences; gray bars – clinical sciences; closed bars – life and clinical sciences.

Table 1.

Publication output of clinical and life scientists at research institutions in Split, Croatia

Research institution	No. of publications
School of Medicine	310
Faculty of Natural Sciences, Mathematics, and Kinesiology	21
Mediterranean Institute of Life Sciences	5
Faculty of Chemistry and Technology	4
Other	10

Of 350 publications from Split, 165 (47%) were authored exclusively by scientists from research institutions in Split, while the remaining 185 (53%) publications were a result of collaborations with other Croatian institutions (75 [21%] publications), international research institutions (75 [21%] publications), and combined international and Croatian collaboration (35 [10%] publications). The mode number of authors per article was 6 (55 publications) (Figure 2).

Figure 2.

Co-authors on publications authored by clinical and life scientists from research institutions in Split, Croatia, 2000-2006.

Of 185 joint publications, majority resulted from collaborations with scientists from Zagreb, the Croatian capital, followed by those from Germany and the USA (Table 2).

Table 2.

Most frequent collaborations between scientists from Split, Croatia, and researchers from other Croatian and international institutions

Croatian city or foreign country	No. of collaborative publications
Zagreb (Croatia)	86
USA	25
Germany	25
Rijeka (Croatia)	13
UK	13
Osijek (Croatia)	12
Bosnia and Herzegovina	12
Slovenia	10
Italy	9
France	8
Norway	8
Israel	7
Canada	5
Sweden	5

The articles from Split institutions were published in 191 different journals. In the following five journals, the authors from research institutions in Split published more than five articles: the Croatian Medical Journal (72/350), Military Medicine (16/350), Annals of Saudi Medicine (9/350), Archives of Medical Research (7/350), and International Journal of Cardiology (5/350). The annual JCR impact factor of these journals for 2006 was 0.825, 0.747, 0.360, 1.275, and 2.234, respectively. Twenty percent of articles (72/350) were published in the Croatian Medical Journal. Of these, majority dealt with clinical medicine topics (48/72; 67%), and other articles reported basic science results (12/72; 16,5%) or other subjects (12/72; 16.5%).

The annual impact factors of evaluated publications continually increased from 2000 (average IF = 2.03) to 2006 (average IF = 2.89) (Figure 3A and 3B).

Figure 3.

(A) Average annual Journal Citation Report impact factor of journals where clinical and life scientists from Split published their research findings from 2000 to 2006. (B) Impact factor histogram.

Average relative impact factor of all publications in one year, expressed as percentile, fluctuated between the lowest 36th (year 2001) and the highest 50th (year 2005). Average percentiles of publications by clinical and life scientists from research institutions in Split increased over the 6-year study period (Figure 4A and 4B).

Figure 4.

(A) Average percentile of journals where clinical and life scientists from Split published their research findings from 2000 to 2006. (B) Percentile histogram.

The most common JCR Subject Categories in which the analyzed cohort of scientist published from 2000-2006 was “Medicine, General and Internal,” followed by “Oncology,” and “Pediatrics” (Table 3).

Table 3.

Journal Citation Report (JCR) Subject Categories which indexed publications from clinical and life scientists from Split, Croatia

JCR Subject Categories	No. of articles
Medicine, General & Internal*	110
Pediatrics; Oncology	22
Biochemistry & Molecular Biology	18
Pharmacology & Pharmacy	16
Cardiac & Cardiovascular Systems; Medicine, Research & Experimental; Sport Sciences	14
Obstetrics & Gynecology	12
Public, Environmental & Occupational Health	11
Cell Biology; Genetics & Heredity; Surgery	10
Endocrinology & Metabolism; Hematology; Immunology; Medicine, Legal; Physiology; Urology & Nephrology	8
Clinical Neurology; Infectious Diseases	7
Anatomy & Morphology; Biophysics; Microbiology; Peripheral Vascular Disease; Transplantation	6
Biology; Engineering, Biomedical; Otorhinolaryngology	5
Anesthesiology; Chemistry, Medicinal; Developmental Biology; Multidisciplinary Sciences; Neurosciences; Plant Sciences; Radiology, Nuclear Medicine & Medical Imaging; Reproductive Biology; Toxicology	4
Biotechnology & Applied Microbiology; Emergency Medicine; Gastroenterology & Hepatology; Medical Laboratory Technology; Ophthalmology, Pathology	3
Acoustics; Chemistry, Applied; Chemistry, Multidisciplinary; Environmental Sciences; Food Science & Technology; Psychiatry; Rehabilitation; Respiratory System; Rheumatology	2
Agriculture, Multidisciplinary; Agronomy; Allergy; Biochemical Research Methods; Chemistry, Analytical; Chemistry, Organic; Dentistry, Oral Surgery & Medicine; Dermatology; Ecology; Evolutionary Biology; Geriatrics & Gerontology; Horticulture; Marine & Freshwater Biology; Mycology; Orthopedics; Virology; Zoology	1

*The category Medicine, General & Internal includes 72 articles (65.5%) published in the Croatian Medical Journal.

Assessment of scientific performance at the University of Split School of Medicine (2002-2006)

We analyzed in more detail the scientific output of clinical and life scientists at the University of Split School of Medicine, as this institution produced 89% of all articles in the studied period. This was done in parallel with assessing the performance of projects receiving grants from the government, so we encompassed the period from 2002 to 2006, when the last grant cycle in Croatia was completed.

The number of articles indexed in Clinical Medicine and Life Sciences sections of the Current Contents that were published by scientists from the School of Medicine increased from 30 in 2002 to 68 in 2006. When the number of published articles was divided with the number of scientists with a PhD in a given year, an increase in the number of articles per scientist was observed, although this number was still relatively small (from 0.29 to 0.50 articles annually). When the number of articles was divided with the number of School teachers (with and without a PhD degree), the situation was even worse, ranging from 0.1 to 0.16 published articles per teacher annually.

The analysis of 12 MSES grants given to researchers from the University of Split School of Medicine from 2002 to 2006 showed that € 2.3 million was given to the 12 principal investigators through contract grants, equipment grants, and salaries of junior researchers in the studied period (Table 4).

Table 4.

Budget of government-funded research projects in the 2002-2006 grant cycle*

Grant. No.	Annual grant budget	Total grant budget	Junior researchers	Equipment budget	Total
0216001	8144	40 722	31 629	42 623	114 973
0216002	13 574	67 870	154 584	143 900	366 354
0216003	6787	33 935	144 782	89 931	268 648
0216005	21 718	108 592	0	86 384	194 976
0216006	23 755	118 773	179 197	203 429	501 399
0216007	16 968	84 838	0	62 296	147 134
0216009	5430	27 148	22 547	92 796	142 491
0216010	1357	6787	24 685	0	31 472
0216011	6787	33 935	44 513	51 178	129 627
0216012	10 859	54 296	93 367	42 567	190 230
0216013	5430	27 148	26 100	45 914	99 162
0216014	4072	20 361	71 565	0	91 927
Total	124 881	624 406	792 970	861 017	2 278 393

*Budget in Euros (conversion rate, € 1 = HRK 7.367).

Those 12 principal investigators published a total of 78 original research articles (or 119 when articles shared by two or more grantees were calculated). Only 34% of articles (41/119) resulted from the research proposed in the grant application. The range of articles published on the subjects proposed in grant applications was 0-8 (Table 5). The overlap range was 0%-78%, because one of the grantees reported the same 7 out of 9 (78%) publications that were already reported by another principal investigator in the School during the observed period.

Table 5.

The Current Contents (CC) publications of principal investigators of the 12 research projects receiving government grants at the University of Split School of Medicine from 2002 to 2006

Grant. No.	Research field	No. of articles						Average impact factor (percentile)
		Croatian Scientific Bibliography	published in CC	project-related	research field-related	unrelated to project	overlaps (%)
0216001	neuroscience	13	9	3	8	1	2 (22)	2.720 (74.6)
0216002	neuroscience	13	10	6	7	3	2 (20)	1.453 (46.2)
0216003	neuroscience	17	10	0	0	10	1 (10)	0.888 (44.9)
0216005	microbiology	13	9	3	9	0	0	2.747 (50.3)
0216006	physiology	23	16	8	13	3	8 (50)	2.413 (65.7)
0216007	physiology	21	15	0	7	8	7 (47)	2.400 (72.2)
0216009	genetics	14	10	6	7	3	5 (50)	2.461 (47.9)
0216010	genetics	4	6	3	3	3	2 (33)	1.162 (36.8)
0216011	genetics	12	9	3	3	6	7 (78)	1.342 (47.5)
0216012	pharmacology	5	6	1	5	1	2 (33)	1.951 (56.6)
0216013	biochemistry	9	9	3	7	2	5 (56)	1.448 (41.7)
0216014	oncology	5	12	5	10	0	1 (8)	1.730 (34.9)
Total		149	119	41	79	40	42 (35)	/
Average		12	10	3	6	3	/	1.950 (51.5)

Some studies had the same research field as proposed in the grant application, but their subject was different than proposed for the grant. In total, there were 79 publications on subjects related to the grant or its research field and 40 articles on subjects not related to the grant or its research field – mostly collaborations with clinicians or medical social sciences (Table 5). When we examined the publication types, all the publications related to grants were original research articles.

When data on expenditure were related to the number of articles (financing of the grants divided with the number of distinct articles from research field), the average cost per article was € 29 210 (€ 2.3 million /79). Analysis of funding source disclosure showed that in 45% of 119 research articles (53/119), sources of funding were not indicated at all, while the remaining 56 articles listed MSES grant (48/119), combination of MSES grant and foreign grants (10/119), or foreign grants exclusively (8/119) as a funding source (web extra table).

Discussion

The annual number of publications of clinical and life scientists from research institutions from Split, Croatia, tripled from 2000 to 2006, and the average impact factor of journals where these articles were published also increased. The majority of these articles were published by scientists affiliated with the University of Split School of Medicine, and the number of articles increased independently of the number of researchers. The fact that the number of articles and their quality are increasing is encouraging, given the meager investments in science by the government and, on rare occasions, by private companies (1).

Our results indicate that clinical and life scientists from Split collaborate more with scientists from Germany and the USA than from Croatian Universities of Rijeka and Osijek. This may be explained by the fact that many Croatian scientists worked a period of time abroad as postdoctoral fellows, which helped them to establish collaborations with foreign scientists (11).

The finding that 20% (72/350) of the analyzed articles were published in the Croatian Medical Journal shows the importance of this journal and its educational mission for the Croatian academic community (12). As much as 67% of the articles in the Croatian Medical Journal were published on clinical subjects, which may indicate that our basic scientists are more successful in publishing their articles in journals specialized in their research fields. Or, simply, since Croatian Medical Journal is indexed in the “Medicine, General and Internal” category, it is a more appropriate journal for clinicians.

Five publications in our data set were attributed to the MedILS (13-17). Scientific work at this institution began in 2007, so they were not able to yield original publications between 2000 and 2006. The authors of these five publications cited dual affiliation.

Some of the articles published under the name of an institution from the University of Split may have been published by authors working in institutions to which they are not actually affiliated, but this is difficult to estimate if the source of funding is not indicated.

The scientometric indicators were used not only for the evaluation of publication patterns of clinical and life scientists from Split, but also for the performance review of the projects receiving government grants. The portfolio of grantees' publications was very diverse. Currently, all those publications can be listed in Croatian Bibliography Index from where the MSES retrieves the data for grant performance evaluation, which may not be appropriate for quality assessment, since not all articles are published on the grant subject.

The rules for the inclusion of a publication into the Croatian Bibliography Index are vaguely defined, which leaves a possibility for a principal investigator to include publications that are not related to a specific research project funded by the MSES.

Besides performance review of the projects receiving government grants, we also wanted to compare the financial input with the scientific output. However, grantees often fail to include information on the funding in their publications, so any kind of calculation is biased. Biyani and Joshi (18) have recently calculated the cost of one SCI research article in Indian state-sponsored research and reported that one article in India is worth about US $226 000. In the United States, academic researchers produced an average of 4.0 publications per US $1 million of academic research and development in 2003, compared with 7.3 in 1993 (19). However, we do not have detailed information on what the authors of these indicators used as inputs, so we do not know if those costs included the salary of principal investigators, all staff, and overheads.

Publication counts are important not only for contracting research grants, but also for the employment of junior researchers and getting equipment grants. However, our results show that all the grantees got both junior researchers and equipment grants regardless of their research performance.

It is important to emphasize that government-funded research projects, equipment, and junior researchers are not the only sources of funding of Croatian scientists. We did not include the salary of faculty and technicians or the cost of overheads in our calculations. Laboratory technicians are often shared between several principal investigators, so they cannot exclusively be attributed to the cost of one research project. Likewise, it is difficult to estimate the percentage of faculty salary received for research activities, as all faculty members have both teaching and research duties. Although we did not include these two financial inputs into our calculations of overall research costs, they certainly increase the cost of research articles.

Considering the slow pace of publishing in science, it is highly likely that these principal investigators will publish more results from government-funded research projects in the near future. Also, one must take into an account that all principal investigators in this School have a heavy teaching load. It is known that faculty who devote more time to teaching have lower research output (20-23). Other reasons why grantees do not produce articles on the subject from grant application may be due to the fact that their grants were unfeasible and that the poor grant evaluation did not detect this, or that there was no proper grant evaluation as the grant cycle unfolded. It is also possible that the preliminary results caused the grantees to change the direction of the research, resulting in an article related to the research field but not to the grant subject.

Another significant result was a rather high overlap between the 12 grantees of the University of Split, School of Medicine. While this is a sign of great collaborative practices of the School, it also may create a problem when these jointly produced articles are attributed to each grant separately. This may mask the reality, as observation of individual grants will show that the publication output was significant, when in fact the same publication was presented as a result of several grants. This can also be explained by the fact that grant amount is not large and that only by collaboration and putting resources together these scientists can produce publishable research.

Our results also demonstrated that current evaluation practices of grant performance review were inadequate. There is no standard format in which grant reports are supposed to be written and there are no regular intervals in which the grant report was supposed to be submitted. Also, there are no consequences for not writing a grant report, for poorly written report, for financial mismanagement of the grant money, or for poor scientific output of the principal investigator. Neither there are independent audits from the MSES to check whether the financial grant reports are accurate.

This lack of rules, audits, and consequences leaves plenty of room for financial mismanagement of the state finances and rewarding of scientific inactivity. Therefore, we propose a series of recommendations for future evaluation of state-funded research grants. First, the MSES needs to publish a standard form for grant reports, which has to be submitted at the end of the each project year. Failure to file a report or an inadequate report should be followed by a termination of further funding. Second, detailed reports should be corroborated with documents and audits performed by peer reviewers and supported by the MSES. Third, only publications stemming from the research grant subject area should be included in the grant report. Some investigators publish in other scientific areas, and if all publications of a scientist in a certain time period are taken into an account, this can give a false image of grant outcomes. Fourthly, independent Croatian bodies such as the Committee for Ethics in Science and Higher Education should raise awareness of ethical principles in science and discourage inappropriate and false awarding of authorship to authors or institutions that did not make a substantial contribution to a publication (24).

The take-home message is that Croatia and other countries in the region need better evaluation of grants as the proposed research turned out to be unfeasible for some scientists. Also, there is a need for more research in this field, as no publications are available on the productivity of research grants in Eastern Europe. We need clear methodology for evaluating scientists' performance, for which relative impact factor of journals where they publish is great candidate. All this needs to be addressed if Croatia really wants to be a knowledge-based society driven by science.