Quality markers and use of electronic journals in an academic health sciences library

Abstract

Objectives: Patterns of use of electronic versions of journals supplied by an academic health sciences library were examined to determine whether they differed from patterns of use among corresponding print titles and to relate the applicability of print collection development practices to an electronic environment.

Methods: Use data supplied by three major vendors of electronic journals were compared to reshelving data for corresponding print titles, impact factors, and presence on Brandon/Hill Lists.

Results: In collections where one-click access from a database record to the full text of articles was possible, electronic use correlated with print use across journal pairs. In both versions, Brandon/Hill titles were used more frequently than non-Brandon/Hill titles, use had modest correlations with journals' impact factors, and clinical use appeared to be higher than research use. Titles that had not been selected for the library's print collections, but which were bundled into publishers' packages, received little use compared to electronic titles also selected in print.

Conclusions: Collection development practices based on quality and user needs can be applied with confidence to the electronic environment. Facilitating direct connections between citation databases and the corresponding journal articles regardless of platform or publisher will support scholarship and quality health care.

INTRODUCTION

Electronic versions of traditional journals are no longer news in academic health sciences libraries. Since the introduction of end-user searching of MEDLINE in the early 1980s, electronic formats and services have achieved a certain dominance in library activities. Every aspect of library service from bibliographic instruction to hours, space, facilities, and equipment has been affected.

These have been exciting, exhilarating times, and librarians have embraced these opportunities. Librarians were not pushed in: they jumped. The Kornhauser Health Sciences Library of the University of Louisville (UofL), like many others, began by adding electronic versions of traditional print journals to its collections in modest numbers using traditional collection development methods. Kornhauser librarians took advantage of packages of titles that, in print, had proved to have high use and high demand. “Free” electronic offerings bundled with print subscriptions were also quickly adopted. The collection development and library materials budget landscape quickly changed when publishers and aggregators provided attractive financial incentives for libraries to purchase access to entire product lines, regardless of whether the library would have selected all the titles using traditional collection development practices. This phenomenon has been somewhat derisively called the “big deal.” At UofL, little additional funding was obtained to provide electronic journals, but user acceptance and demand was immediate. Acquisitions continued exponentially, funded by internal reallocation.

Although it did not seem like it at the time, from today's economic perspective, UofL's budget situation at the end of the century allowed a certain freedom for librarians “to jump right in” and provide both paper and electronic forms of journals, and they were able to supply the critical mass of electronic journals that rapidly won users, for better or for worse. These heady times did not allow much time for reflection, often leading UofL librarians to wonder if they were being truly proactive rather than simply reactive. At the beginning of the new century, however, market forces are requiring reevaluation of some of the choices that were made.

Is journal selection still an important function for libraries? Do traditional collection development principles still apply in the world of the “big deal?” Has ease of access through one click from citation to electronic full text or convenient availability of broadened collections changed library users' behavior? Are the choices users make in an electronic environment different from those they make in the print environment? Is quality being sacrificed for convenience—or is convenience an aspect of quality?

The good news is that the electronic environment provides tools for quantitatively tracking use of library services that were not available with library systems that evolved to manage paper-based collections. Computers count transactions very well, whether by circulation data tracked by local systems or use data provided by some library vendors.

An initial examination of the data supplied by major vendors to UofL appeared to show that electronic use has been considerably higher than print use for titles held in both print and electronic versions. It also suggested that a direct link between MEDLINE citations and journal full text (i.e., one-step access) was taking users to those journals more frequently than to journals that did not have one-step access. Librarians in Louisville were concerned that ease of access trumped quality for library users. How does use of journals in an electronic environment relate to external measures of quality, utility, or, at least, notoriety? How useful are traditional selection processes in an electronic environment? Vendor-supplied use data for journals from three electronic collections were examined and compared to quality markers and to reshelving data gathered for the corresponding print titles to see what could be learned.

LITERATURE REVIEW

Users have embraced electronic journals enthusiastically, and not just in Louisville. Rogers's surveys at Ohio State University [1] indicated dramatic increases in the numbers of faculty and students who used electronic journals over three years, 1998 through 2000. Several factors appeared to be at work: a growing awareness of the potential for reaching journals online, direct links from citation databases to full text, and growth in the “critical mass” of electronic collections. These data also showed a decrease in the use of print journals. Biological and medical sciences had the highest users in both categories, an observation confirmed by Tenopir's survey data [2].

Reshelving studies at the Library of the Health Sciences–Peoria, University of Illinois at Chicago, by De Groote and Dorsch [3] indicated that, as electronic collections grew, use of journals in paper formats decreased, both for journals held only in paper and journals held also in electronic form. Their recent study employing survey data has confirmed their users' preferences for electronic access to journals [4]. This and a decrease in interlibrary loan requests suggested that users might be compromising quality for the convenience of electronic journal access. Sathe and colleagues at Vanderbilt University [5] provided a substantial review of work on user preferences for electronic formats and a persuasive agenda for future studies on patterns of use that will help librarians draw conclusions to guide development of collections and services to support their missions.

Because users of electronic journals can bypass library services like catalogs or lists, libraries depend on vendors for counts of actual use. Vendor-supplied data have been subject to criticism on library email discussion lists for being self-serving, inconsistent, ill-defined, and scarce. These complaints bring to mind the old non sequitur concerning airline food: it's terrible, and there's not enough of it! Mercer [6], Blecic and colleagues [7], and Luther [8] have discussed issues concerning vendor-supplied data. Though standards for better statistics are still evolving [9–11], more journal publishers and aggregators are providing numbers for library use. Consistent and reliable vendor data sets have the potential to enable librarians to understand user behavior better than ever and to help inform decision making for libraries. Despite the current limitations, librarians have begun to experiment with ways to gather this information, to discern meaning in it, to discover patterns of user behavior, and to use it to inform decision making. Morse and Clintworth [12] used both vendor-supplied transaction data (Ovid) and data collected during reshelving to show that users at the University of Southern California accessed an electronic version of a journal more than ten times as often as the corresponding print journal. They also ranked use for paper and electronic formats and found that 60% of the use came from the top 20% of the titles in each case. They concluded that users of electronic journals were using essentially the same journals they would have used in print.

In making selection decisions, librarians have used a variety of methods to predict which journals will be most useful to their patrons. Citation data have long been used in libraries as one quantitative indicator of utility. In 1994, Kelland and Young [13] reviewed a number of studies that examined whether the frequency with which a journal's articles are cited could serve as an objective measure of the journal's usefulness to library clients. While these studies demonstrate that the relationship between citations and literature use is very complex and certainly multifactorial, citation data have been shown to be a valuable factor in collection management decisions and a modest predictor of library use.

ISI facilitates the use of citation data by publishing impact factors for the journals it selects for analysis. The impact factor (IF) is defined as the ratio between the number of articles from a given journal published in the previous two years and cited by journals analyzed by ISI and the number of articles published by that journal during those same two years [14]. The implication is that journals with a higher IF, in other words, those cited more frequently by authors, have more impact on a field, more prestige in a field, and more utility for the reader. In 1998, Tsay [15] demonstrated a significant correlation between use as measured by reshelving data and both citation use and IF for journals in clinical medicine and life sciences. Saha and colleagues [16] recently tested the relationship in internal medicine journals and found strong correlations between IF and physicians' subjective ratings of journal quality. Duerenberg [17] and Smith [18] have discussed use of IF as one criterion for deselection of journals in an academic medical library. Though there are many caveats regarding the use of IF [19], it has, in the absence of other quantitative measures, come to be used as one marker for quality.

Another indicator for utility commonly used in health sciences libraries is the “Brandon/Hill Selected List of Print Books and Journals for the Small Medical Library.” This bibliography was first created by Alfred N. Brandon in 1965 to guide hospital libraries in collection building. The list includes a core collection of medical journals listed alphabetically and by subject area. It is supplemented by similar lists for nursing and allied health fields [20]. Because of its intended use by hospital libraries, the list has a clinical emphasis. The Brandon/Hill list has long been considered the expert bibliography for health sciences libraries.

In 2002, Lee and colleagues assessed the methodological quality of 243 articles and compared their score with a number of journal characteristics. They found presence on the Brandon/Hill list to be a strong predictor of higher scores for methodological quality, along with high citation rates, IF, circulation rates, and low manuscript acceptance rates [21].

SETTING

The Kornhauser Health Sciences Library is a medium-sized academic health sciences library that serves, as primary clientele, the faculty, staff, and students of the schools of medicine, dentistry, nursing, and public health and information sciences of the UofL. The library is part of the university's seven-library system. The other six libraries are located on the Belknap campus, three miles away. All of the libraries' resources, both print and electronic, are available for use by all university faculty, staff, and students. Journals are available on university networks through Internet protocol (IP) address recognition and off campus via proxy server. No cancellations of print materials in favor of electronic materials had been made. The only electronic titles held for which the library did not have print counterparts were acquired as part of package deals.

Links for access to electronic versions of journals were included in the holdings record in the library's Web-based catalog and in a separate list of electronic journals prominently linked from the library's Web page. Links to major aggregators' Web pages were also available. Full text of articles available through Ovid was linked directly from MEDLINE, the Cumulative Index to Nursing and Allied Health Literature (CINAHL), and PsycInfo citations. Access to other electronic full text required the separate step of locating a link to the journal. Various databases, including MEDLINE and CINAHL, had been available through Ovid on university networks since 1995. The availability of electronic full text had been widely announced in newsletters, classes, and meetings—all the normal channels for marketing library services.

In addition to MEDLINE via Ovid, many use PubMed for searching. Studies concerning which users choose PubMed rather than Ovid/MEDLINE and the extent of PubMed use remain to be done. In 2001, PubMed's Linkout feature provided links to articles available through Elsevier's ScienceDirect (SD) and Ideal/Academic Press but not to Ovid articles.

Of the 1,156 electronic biomedical titles to which the library subscribed in 2001, detailed statistics were provided for 506 titles by three vendors, Ovid, SD (Elsevier), and Ideal (Academic Press). Gross numbers of articles accessed in either portable document format (PDF) or hypertext markup language (HTML) appeared to be consistent enough across vendors to make some generalized comparisons. In 2001, most titles were available to the library's users through only one electronic vehicle, so duplication was not a major problem. This study was undertaken with the knowledge that it would be a snapshot taken in a particular library during a time when rapid change in the availability and acceptance of electronic versions of journals was occurring.

All titles for Ovid full-text journals were selected specifically for health sciences users, though they were also available to the entire university community. In 2001, library users had access to 113 current Journals@Ovid titles, including the four biomedical collections, the two nursing collections, the mental health collection, and selected individual titles. The Ovid collections had been selected and packaged by Ovid from high-profile journals from a variety of publishers. Coverage for these titles was mostly 1995 continuing through 2001, with Ovid's Core Biomedical Collection going back to 1993. Ovid full text had been available to UofL users since 1998.

In 1999, the university libraries purchased access to all electronic titles in Elsevier's SD collection, including many titles not held in print by any of the seven libraries in the system. For this study, the authors selected biomedical titles (i.e., those journals that fit the collection development policy of the Kornhauser Health Sciences Library), whether or not they had been selected for the print collection: 236 titles, a mix of clinical and research titles. The Kornhauser Health Sciences Library had print subscriptions for 177 of these. During the time for which these data were gathered (2001), most titles were electronically available from 1993 through 2001.

The university acquired access to the Academic Press collection via Ideal in October 2000. Titles matching the collection development policies of the health sciences library were selected for analysis. During the period under consideration, most of these titles had back-files available from 1993 through 2001.

METHODS

A spreadsheet was constructed of the biomedical titles from the Ovid, SD, and Ideal collections to which UofL patrons had current electronic access throughout 2001. The presence of a title on the “Brandon/Hill Selected List of Print Books and Journals for the Small Medical Library” [22], “Brandon/Hill Selected List of Print Nursing Books and Journals” [23], or the “Brandon/ Hill Selected List of Print Books and Journals in Allied Health” [24] was noted. Impact factors for those journals analyzed by ISI in the science edition of Journal Citations Reports (JCR) [25] were recorded.

Print use (pUse) for each title held in print was derived from circulation records for the calendar year 2001. Item barcodes were routinely scanned as bound volumes were replaced on shelves. Tick-marks were made on unbound issues, the number of tick-marks was entered as circulation data when those issues were bound, and pUse represented the sum of these data for each title. This method certainly underestimates use, because patrons reshelve journals, in spite of entreaties not to, and because the number of articles patrons look at when they take an issue or volume from the shelf is not known. Milne and Tiffany have suggested that this method may underestimate use by a factor of 4 [26]. The 2001 circulation figures for print titles included the whole range of the library's holdings for the journal; no attempt was made to adjust these figures to include only the time period covered by the corresponding electronic version. Because the average length of run for the library's print titles was 29 years, compared with the 5 years available in electronic form, some of the pUse count came from journals more than 5 years old. However, Maxfield and colleagues have reported that 90% of a journal's use is from the most recent 5 years [27], which suggests that the period of use of most of the paper journals is close enough to the period available for electronic versions for the purposes of this study.

Use of journals available through Ovid was determined by the figure given in the Web-based Ovid Statistics Report Generator for full-text encounters, by journal title, using the figures for “all types” of articles. SD data came from its Web-based statistics program. For each title, the sum of PDF and HTML articles downloaded was recorded. Data for Academic Press titles were gathered from Ideal's Web-based statistics program. Numbers of Fulltext Downloads were taken from Journal by Journal Statistics for 2001.

Data were normalized for title changes and entered into an Excel spreadsheet. Statistical functions were calculated using Excel and SPSS for Windows (v.11.5) for various populations. Reported correlations are Pearson product moment correlation coefficients (r).

RESULTS AND DISCUSSION

Correlation between electronic use and print use

The use of the average library-provided electronic journal (eUse) and of its corresponding print journal (pUse) and correlations were calculated across corresponding pairs, as summarized in Table 1. The average print journal item, bound or unbound, in this study was replaced on shelves by staff 83 times. The average article in the corresponding electronic set was accessed 223 times. Because the 3 vendors may use somewhat different methods for collecting and reporting use, it should be remembered that comparisons of eUse among the collections is speculative and is done only to provide a baseline. With that caveat, it appears that the collection designed for a clinical audience (Ovid) was most highly used in both print and electronic versions and that the ratio of eUse to pUse was highest for that collection—almost twice that of the SD and Ideal collections, if consistency of reporting among the 3 vendors can be assumed. Correlations indicated that pUse was a reasonable predictor of eUse for each collection.

Table 1 Average use per title of journals with both paper and electronic ac cess

While the ease of one-click access between citations and full text is certainly a factor, the quality of the different collections may vary. To determine if the collections were of similar quality, impact factors were examined together with frequency of use in both the electronic and print environments.

Correlation between impact factors and use, electronic and print

The average use of titles held in both print and electronic formats for each collection, among those journals for which ISI provides IF, has been calculated, and correlations with IF are summarized in Table 2. Table 2 represents a subset of the journals in Table 1 (i.e., those titles for which an IF has been calculated). The correlations between print and electronic use remain similar to those in Table 1. Selection of articles from the Ovid collection correlates modestly to IF for both print and electronic use, but little, though positive, correlation exists between IF and use of SD or Ideal titles, either in print or paper.

Table 2 Use of titles held in both print and electronic versions relative to impact factor (IF)

The range, average, and median for IF for the three collections suggest that differences among the collections should be considered when drawing conclusions from these results, and these differences illustrate the caveats for quantitative use of IF. While a thorough analysis of these factors is beyond the scope of this paper, they help characterize the three collections. The range of IF for Ovid titles is greatest, and the average and median IF are highest. It should be noted that most of the titles in the Ovid collection selected for UofL have been selected twice: first by Ovid, for its marketing purposes to create a core collection of high-demand general interest titles (drawn, of course, from those publishers that would cooperate). Then, the Ovid collection was further supplemented by choices made by this library to create a core electronic collection for its own clinical users. This method of selection is very different from the acquisition of the complete offerings of a single publisher. The SD collection, under the terms of the university's agreement, consisted of the whole range of publications available through SD, ranging from high-profile titles that many in a research university environment would be interested in to publications that only a few would be interested in. The range of IF illustrates this. The Ideal/Academic Press titles analyzed here tended to be specialty titles, well respected within small fields, but of interest to small groups of users in specialty areas that might not be represented on every campus. These kinds of titles tend to have lower IF than less specialized titles. While the range of IF is smaller for these titles, the median IF is about the same as SD.

Comparison of use of Brandon/Hill and non-Brandon/Hill titles

The Brandon/Hill lists have been developed to identify those journals most important for clinical care, and a journal's presence on those lists can be construed as a proxy both for quality and value in supporting clinical care. At the time of this study, the library's electronic collection did not include any of Ideal's Brandon/Hill titles.

If quality is important to electronic journal users, Brandon/Hill titles would be expected to be used more frequently than non-Brandon/Hill titles, when ease of access is similar. Table 3 shows that among Ovid journals, the users did access considerably more articles from Brandon/Hill titles than non-Brandon/ Hill, in even higher proportion than in print.

Table 3 Comparison of use of Brandon/Hill and non-Brandon/Hill titles

Table 3 also shows that, while use of paper Brandon/Hill titles is similar for the Ovid and SD titles, electronic use of Brandon/Hill titles is more than four times higher for Ovid titles than for SD titles. In fact, SD Brandon/Hill titles were used less often than the corresponding paper titles. Among the non-Brandon/ Hill titles, use of electronic versions in both collections is similar.

The findings indicate that patrons looking for information in clinically oriented journals used electronic journals with one-step access from citations (Ovid) more frequently than journals that required a separate lookup (SD). However, users of print journals also used the paper journals of the Ovid collection more frequently than they did the print versions of the SD collection, so electronic convenience is certainly not the only factor in their selections.

Use of Brandon/Hill Ovid titles with impact factors

To determine whether users were as discriminating in their journal choices in the electronic environment as they were in a print environment, the use of Brandon/ Hill journals in the library's electronic collections was compared to their impact factors. Not all Brandon/ Hill titles are analyzed by ISI and given impact factors, so fewer titles enter into this analysis. Of the eleven SD titles that met these criteria, four were also in the Ovid collection, so that sample was not only small, but confounded, and so only the Ovid collection was used for this analysis. The results are summarized in Table 4.

Table 4 Use of Brandon/Hill Ovid titles with impact factors

Of the 46 titles, the Pearson correlation for use with IF was 0.77 for print versions and 0.70 for electronic versions. Apparently, users viewed more articles from journals with higher impact factors, and the pattern in the electronic environment seems to be similar to the pattern in the print environment. (It should be noted that while Tsay reported a Pearson correlation of 0.16 for the less selective list of 264 clinical medicine titles she examined, she found a correlation of 0.34 among the 50 most frequently used mixed scientific titles in her study [28].)

One-step access from PubMed to basic research titles

Support of clinical care is not the only mission of the academic health sciences library. While recognizing that research takes place in all disciplines on the health sciences campus, it runs on a continuum ranging from research that supports health care, to clinical research, to basic sciences or “bench” research. At the risk of pigeonholing user behavior too neatly, the authors identified a subset of journals that would be of broad research interest across biomedical fields to examine patterns of use among bench researchers in contrast to those of clinical researchers. This subset, hereafter referred to as “research” titles, was defined as journals available through Ovid, SD, and Ideal that were identified by ISI's JCR category lists as titles corresponding to basic science fields represented at the university's health sciences center. The JCR categories used were biochemistry and molecular biology, cell biology, multidisciplinary sciences, and physiology. Also included were those titles classified as “medicine, research and experimental.” Only one of these titles (Journal of Clinical Investigation, via Ovid) was also on the Brandon/ Hill list. The results are summarized in Table 5.

Table 5 Use of research titles

Reference desk experience and other anecdotal evidence suggest that these research scientists are mostly PubMed users. If this assumption is correct, the data in Table 5 suggest that one-click access is an important factor in the use of electronic journals for bench researchers, as it appears to be with more clinically oriented users. SD and Ideal articles were available through PubMed's Link Out feature, while access to Ovid titles required additional steps for PubMed users. The ratio between pUse and eUse was not nearly as high for the Ovid research titles as it was for the Ovid Brandon/Hill titles, in spite of the fact that this small group of research titles had higher average IF; that high-profile, cross-disciplinary titles (e.g., Science, Nature, Journal of Clinical Investigation, and Journal of Cell Biology) were included; and that, at the time, the university libraries provided no alternative electronic access to them. Perhaps paper copies of these journals were so nearly ubiquitous in the research environment that the embargoes associated with some of these journals and lack of one-step access for PubMed users made electronic access to these journals relatively unattractive to researchers.

The modest correlations between eUse and pUse indicated that print use was a reasonable predictor of electronic use for the research subset in each collection, as it was among the more clinical Brandon/Hill titles.

There was no significant correlation between IF and either print or electronic use for SD and Ideal titles. This is consistent with the observation by Stankus and Rice [29] that print use correlates with IF only for titles grouped by discipline and with the “nitchiness” that characterizes the Ideal and SD titles identified by the research definition.

Use of selected and not selected titles

Use of electronic versions of journals that had not been selected by the library for the print collection (eOnly), but were bundled by the publisher or aggregator with packages of electronic journals, was compared to eUse of those that had been selected individually through standard collection development practices. Data in Table 6 show that, among all three sources, the mean eUse of the not selected titles was considerably lower than the mean eUse of journals that had been selected individually to meet the needs of this library's users. Not selected titles in the Ovid collection were used much more frequently than titles in the other collections, suggesting that the one-click access increased the appeal of these titles over those requiring multiple steps.

Table 6 Use of selected and not selected titles

CONCLUSIONS

These data and analyses indicate that, within collections, users of electronic versions of library-provided journals accessed clinical titles more frequently than research titles, that they used Brandon/Hill titles more frequently than non-Brandon/Hill titles, and that they used Brandon/Hill titles in the Ovid collection in proportion to their IF.

While people used electronic titles in general more often than the corresponding print version, they showed the same relative behavior in the electronic environment as in the print environment, especially when multiple steps were not required to locate the electronic journal. One-click electronic access from citation to article is important to users and an important factor in electronic journal usage. At times, users find that the best article is the most convenient article, and they choose to end their search at that point. However, the data presented here indicate that, in the one-click environment, users tend to remain selective in choosing their articles in ways that can be predicted from quality indicators used for traditional library collection development practices. Journals highly used in the print environment are also highly used in the electronic environment. Impact factors and presence on the Brandon/Hill lists, predictors of high use in paper, also predict high use in electronic forms.

It appears to be reasonable to expect that any suspected lack of discrimination in an electronic environment will be mitigated as access to articles across vendors' or publishers' platforms is facilitated. The case can be made that libraries, publishers, and vendors will support the interests of scholarship and quality health care in general, as well as making users' lives easier, by facilitating direct connections between citations and corresponding journal articles (e.g., open URL initiatives) regardless of platform or publisher.

This study demonstrates that collection development practices that built collections designed to meet the clinical and research interests of a library's users are independent of medium. Collection development practices that created collections designed to fit a particular institution's needs can be applied with confidence to the electronic environment, and journal use statistics from the paper environment can be used with confidence to inform collection decisions. Titles that are, from the library's perspective, of little use to the library's users are little used, whether the electronic or paper version. Library resources should be directed toward those titles that fit the needs of the library. Suspicion that the “big deal” may not be the “best deal” for individual libraries is supported.

The data presented in this study represent one library's experience in one year at a time when electronic use was already affecting print use. Longitudinal and multi-institutional studies are in order to determine trends over time and to determine whether these results can be generalized to other settings. Assumptions made about user behavior, such as those made about MEDLINE use via PubMed and Ovid should be tested, and the results used to assess acquisition and training priorities. Vendor-supplied data should be compared to local measures such as clicks from library Web pages. Increased availability of vendor-supplied data, combined with studies of user behavior will lead to better evidence for better library practice.