Reading factor: a new bibliometric criterion for managing digital libraries

INTRODUCTION

Although various methods are used to evaluate biomedical journals, there is no standard reference for fully assessing the impact of a journal in clinical medicine [1]. The utility or influence of a given paper on diagnosis or therapeutics has not been clearly defined. This study presents a novel mode of evaluation. In the research literature, an acknowledged criterion of value is the degree of influence on secondary papers. A citation by other articles is accepted as a fair estimate of the value or importance of a publication [2]. The impact factor (IF), published in the Science Citation Index (SCI) Journal Citation Reports (JCR), is based on a bibliometric analysis of science journals in the ISI database [3]. Although it has been criticized [4–8], IF is the most common bibliometric criterion for ranking and evaluating biomedical journals [9]. As IF represents, by its nature, an indirect criterion, a direct measure of journal use should be considered. The number of times a publication is consulted is an objective criterion of use. The recent availability of digital versions of major biomedical journals presents an opportunity to directly measure their electronic consultation rates.

This paper describes a way to measure the electronic consultation rate that the authors have termed the reading factor (RF), in direct correlation to the impact factor. RF measures the interest in a journal within the limit of a given readership. It is a direct measurement of journal use that is related to both clinical medicine and library or information science. Results of the observed distribution of RF for the year 1998 and IF for 1997 are shown in Table 1. The Rouen University Hospital (RUH) medical digital library was created in 1997. This library allows all 304 RUH senior physicians to access MEDLINE, forty-six electronic full-text journals, and some selected electronic books from their offices at no charge [10].

Table 1 Impact factor (IF) versus reading factor (RF) based on forty-six biomedical journals

METHODS

Electronic full text has been provided by Ovid®, a commercial company, via the RUH intranet since 1997. These journals are offered as packages known as Biomedical Collections volumes I, II, and III, which contain fifteen, fifteen, and sixteen journals, respectively. They have been available at RUH since June, September, and December 1997, respectively. The journals in each package are selected by Ovid based on the coverage of major biomedical specialties, journal impact factors, and agreements with publishers. Table 1 includes all forty-six journals.

The number of electronic consultations is incremented each time end users click on a hyperlink to open individual publications. To obtain a standardized measure of the electronic-consultation rate and to avoid an institution-size effect, the RF is defined as the ratio between the number of electronic consultations of an individual journal and the mean number of electronic consultations of all the journals studied or as the following equation:

where C_j is the number of electronic consultations of journal j, and N is the total number of journals available in the database. The normalization of the reading factor has also been performed, because ISI journal impact factor is a similarly normalized value. Thus, a value of 1 represents an average consultation rate, while a value greater than 1 represents a higher-than-average consultation rate. The normalized reading factor is defined (data not shown) as the ratio between the number of electronic consultations and the number of articles of a particular journal.

The distribution of RF was assessed for the year 1998, the first calendar year with full electronic availability of the forty-six journals listed in Table 1. RF and type of publication (article, letter, review, case report, editorial, or miscellaneous) were automatically extracted from log files using Ovid software. Finally, it should be noted that hard copies of the forty-six journals were available two to three months before the release of the electronic versions in 1998 at RUH. The RUH digital library was primarily devoted to physicians but was also available to residents and medical students upon request.

Data on IF from JCR were retrieved from a 1997 CD-ROM edition. RF for the year 1998 and IF for the year 1997 were the latest information available, when this study was performed. The existence of a correlation between IF and electronic journal use as measured by RF was assessed with the Pearson and Spearman rank correlation coefficients. The Kruskal-Wallis test was used to assess the existence of an association between RF and each Biomedical Collection package. Because electronic journal use varied according to the Biomedical Collection package, a multiple linear regression was used to study the existence of a correlation between RF and IF while controlling for Biomedical Collection package. Analyses were performed using BMDP New System for Windows, version 1.1 (BMDP Statistical Software, Inc.), and StatXact software, version 3.0.2 (Cytel Software Corporation).

RESULTS

A total of 5,007 electronic articles were consulted in 1998 (Table 1). There were 2,349 original articles (46.9%), 1,011 letters (20.2%), 378 review articles (7.5%), 336 case reports (6.7%), 310 editorials (6.2%), and 623 with no specific publication type (12.6%).

Table 1 displays the number of electronic consultations as well as the value of RF in 1998 (in decreasing order) and the value of IF in 1997 for each journal. The mean number of publications electronically consulted per journal was 108.80, while the corresponding median number was 73 (range 7–630). The mean RF was 1 by design, while its median was 0.47. The distribution of RF was skewed with only 19 values above the mean and a wide range of values from 0.06 to 5.78. The Lancet and The New England Journal of Medicine had the highest RF values (5.79 and 4.01, respectively), while Nature and Science had relatively low RF values (0.40 and 0.55, respectively), which was consistent with a university hospital environment.

The mean 1997 IF of the 46 journals under study was 6.0, and their median IF was 3.4 (range 1.4–27.8). The distribution of IF for these 46 journals was also very skewed, with only 12 IF values above the mean and a wide range of values from 1.42 to 27.76. The highest IF values were those for The New England Journal of Medicine, Nature, Science, and The Lancet (27.76, 27.36, 24.67, and 16.13, respectively).

No correlation was found between the 1998 reading factor and the 1997 impact factor of these forty-six journals.

Statistical analysis of correlation between reading factor and impact factor

From these data, the observed Pearson correlation coefficient between IF and RF was 0.3737, which would indicate a positive and significant correlation (P = 0.0105). However, this indication would be misleading, as both the distribution of IF and that of the number of articles were clearly not normal. In fact, this apparent positive correlation was essentially driven by the two major medical journals, The New England Journal of Medicine and The Lancet. The Pearson correlation coefficient for the remaining forty-four journals was estimated at −0.0390, a value non-significantly different from 0 (P = 0.802). These two journals had a highly influential role on the examined correlation, because they had very high RF values (5.79 and 4.01, respectively, the two highest values by far) and very high IF values (see above).

To override these distribution discrepancies and perform a more valid analysis, we reestimated the Pearson correlation coefficient on log-transformed variables and estimated the Spearman rank correlation coefficient on all forty-six journals. We found very close values of 0.1775 and 0.1417, respectively. These values did not significantly differ from 0 (P = 0.238 and P = 0.348, respectively), thus failing to lend any support to the existence of a correlation between IF and electronic journal use as measured by RF. Figure 1 illustrates this startling lack of correlation. The authors also studied the correlation between IF and RF separately for different types of publications (original articles, letters, review articles, case reports, editorials, and other publications) with the Spearman rank coefficient and found no significant correlation.

Figure 1.

1997 impact factor as a function of the rank of 1998 reading factor for forty-six biomedical journals

Finally, as we found a significant association between RF and Biomedical Collection package (P = 0.0051, Kruskal-Wallis test) on the one hand and a nearly significant association between IF and Biomedical Collection package (P = 0.0900, Kruskal-Wallis test) on the other hand, we studied the correlation between RF and IF while controlling for Biomedical Collection package. Using multiple linear regression, we found again no significant correlation between the log-transformed variables IF and RF (P = 0.925).

We found highly significant correlation among the RFs for the years 1998, 1999, and 2000, as pairwise Spearman's rank correlation coefficients were 0.9682 for years 1999 and 2000 (P < 0.0001), 0.8738 for years 1998 and 2000 (P < 0.0001), and 0.8930 for years 1998 and 1999 (P < 0.0001).

DISCUSSION

One of the major advantages of this approach is the availability of results with minimal delay. Electronic access allows a reliable and automatic evaluation of the consultation rate of such documents. It appears to be a more practical version than the manual count of documents or printed copies borrowed from the library [11].

As with its printed counterpart, however, there is no way to determine whether this corresponds to a partial or a complete reading, an effect of curiosity, or even an accidental mouse click. That is why the RF is not indicator of quality but is at best an inferential indicator. To avoid taking into account accidental mouse clicks, a revised version of the RF will count a click if a visitor spends an appropriate amount of time looking at the document (e.g., 30 seconds).

In the Rouen University Hospital, some physicians may still be somewhat reluctant to use this digital library and may prefer printed versions when available or continue to individually subscribe to some journals (internal questionnaire; data not shown). Because a significant level of use of alternate information means persists, RF may provide an accurate estimation of the interest in the digital version of the journal but cannot be considered an indicator of the interest in a publication.

Some novelty effect might exist and might have artificially increased electronic journal use, because we noted that RF values were higher for journals acquired in June 1997 (Biomedical Collection I) than for those acquired afterward (Biomedical Collection II and III). Furthermore, electronic journal consultations increased 63% (3,147 versus 1,926) for the first five months of 1999 compared with the corresponding months of 1998. In contrast, electronic journals without a corresponding printed version available yielded low RF values.

To our knowledge, this is the first study to compare electronic journal use and citation frequency. A few published studies have evaluated the relationship between printed journal use and citation frequency but with somewhat inconsistent results. Tsay investigated the relationship between journal use in a Taipei medical library and journal citation in the biomedical field [12]. The results of Tsay's study showed a significant and positive correlation between frequency of use and IF for all titles, although the estimated Pearson and Spearman correlation coefficients were rather low (0.34 and 0.35, respectively). Tsay also found a significant and positive correlation between frequency of use and IF, when journals that published clinical medicine and journals that published life science articles were considered separately. In contrast, no correlation or only partial correlation was found between journal use and citation patterns in biomedical sciences for studies published in the late 1970s [13–16].

The results of this study suggested that RF provided different bibliometric information than IF. The frequency of use of a journal could therefore be a significant parameter of its interest to readers and could be used as a more relevant marker of a given journal's influence. For instance, in selected specialties within specific readerships, RF tended to reveal differences between journals of equivalent IF. A sharp contrast between IF and RF was observed among three pairs of publications: Chest and Thorax, American Journal of Psychiatry and Archives of General Psychiatry, and American Journal of Surgery and Archives of Surgery (Table 1). RF could thus be regarded as an additional criterion for the selection of electronic journals for hospital physicians.

It is worth emphasizing the results obtained from our analysis for four journals. Two leading generalist medical journals (The New England Journal of Medicine and The Lancet) had the highest ranks of RF (2nd and 1st, respectively) and among the highest ranks of IF (1st and 4th, respectively). On the other hand, two major generalist science journals (Nature and Science) had among the highest ranks of IF (2nd and 3rd, respectively) and among the lowest ranks of RF (32nd and 25th, respectively). If Nature can be taken as almost “twice as good” [17] as The Lancet in terms of IF, the ratio between The Lancet's RF is fourteen times as high as Nature's RF. RF measures the interest in a journal within the limits of a given readership and cannot be taken as an indicator of the journal's quality.

The results of this study are consistent with the fact that most senior physicians at our university hospital are more clinically involved than research oriented. Nonetheless, these results are partial in terms of electronic journals coverage (46 versus over 3,000 now accessible on the Internet) and as based on a single institution in terms of readership.

The rapid increase in journal prices, both electronic and printed, has made the optimization of collection management essential [18]. The results of this study suggest that collection managers would not be able to predict electronic journal use on the basis of journal impact factors alone. Complementing the quality criteria, the RF could become an economic criterion to optimize electronic journal management in academic institutions, as the cost of a click can easily be compared to the cost of interlibrary loan [19]. In the RUH, the average cost of each electronic article was $3.92 in 1998, $2.53 in 1999, and $1.89 in 2000. The RF could also be used at the university level (e.g., over 1,500 electronic journals are available in the Rouen University) or directly aggregated for various institutions by a commercial company (e.g., Ovid using its Website). It can help library decision makers acquire new publisher collections according to previous RFs.

CONCLUSION

The measurement of RF is highly automated and practical. RF is an objective and immediately available criterion of local electronic journal use or interest in a particular electronic journal. Within the limits of a single institution at the beginning of its experience, RF appears not to be correlated with IF. This lack of correlation suggests that RF provides different bibliometric information than IF. RF is a promising economical criterion for local collection management of an electronic library. It should, however, not be used in isolation but should be considered with other quality indicators or indexes of scientific relevance.