Identifying and communicating the contributions of library and information services in hospitals and academic health sciences centers

Abstract

Objective: This article introduces a systematic approach to identifying and communicating the value of library and information services (LIS) from the perspective of their contributions to achieving organizational goals.

Methods: The contributions of library and information services (CLIS) approach for identifying and communicating the value of LIS draws on findings from a multimethod study of hospitals and academic health sciences centers.

Results: The CLIS approach is based on the concept that an individual unit's value to an organization can be demonstrated by identifying and measuring its contributions to organizational goals. The CLIS approach involves seven steps: (1) selecting appropriate organizational goals that are meaningful in a specific setting; (2) linking LIS contributions to organizational goals; (3) obtaining data from users on the correspondence between LIS contributions and LIS services; (4) selecting measures for LIS services; (5) collecting and analyzing data for the selected measures; (6) planning and sustaining communication with administrators about LIS contributions; and (7) evaluating findings and revising selected goals, contributions, and services as necessary.

Conclusions: The taxonomy of LIS contributions and the CLIS approach emerged from research conducted in hospitals and academic health sciences centers and reflect the mission and goals common in these organizations. However, both the taxonomy and the CLIS approach may be adapted for communicating the value of LIS in other settings.

INTRODUCTION

Many difficulties face library and information services (LIS) directors as they attempt to measure and report the value of their units' operations to the larger organization. Determining the value of a unit's operations is traditionally approached quantitatively through the use of techniques such as cost-benefit analysis and return on investment calculations [1]. However, the intangible nature of LIS makes a quantitative evaluation challenging because of the absence of isolated, identifiable products or outcomes. Furthermore, it is often difficult, if not impossible, to disaggregate the causes of specific benefits or cost savings. For example, in clinical settings, the results of LIS services may be combined with other sources of information such as results of diagnostic tests, making it impractical to measure the precise nature of LIS impact. The study conducted by Klein and her colleagues is a rare example of how LIS usage data may be associated with contributions to organizational goals [2]. Although their findings did not establish causal relationships or isolate LIS from other inputs, the researchers found a significant correlation between the timing of mediated search requests for patient cases and length of hospital stays.

Given the difficulties inherent in measuring the value of their services, LIS staff often rely on quantitative data that reflect the unit's usage as well as qualitative and anecdotal evidence that focuses on the perceptions of individual users. Common quantitative measures of use include the number of database searches and counts of reference and circulation transactions. Such data may be gathered at the unit level or as part of a library or information center's participation in collaborative, cross-institutional efforts, such as the Association of Academic Health Sciences Libraries' Annual Statistics report or the Medical Library Association's Hospital Benchmarking Initiative [3, 4]. Data reflecting individual users' perspectives include reports of satisfaction and service quality or estimates of time spent looking for information and “value in use” [5, 6]. These data may be gathered using instruments developed in-house or with standardized instruments such as those used in LibQUAL+ and other cross-institutional studies [7]. A number of studies have systematically documented the impact of LIS use for individual clinicians [8–10]. While this focus on users' perspectives is consistent with a broad interest among LIS researchers, it often fails as a way of communicating a unit's value to administrators within a particular organizational setting because there is no demonstrable link to achieving organizational goals [11].

This study, which was funded in part by the Medical Library Association, was initiated in June 2000. The study's objectives were (1) to determine appropriate methods for measuring the value of library and information services to hospitals and academic health sciences centers (AHSCs) and (2) to identify the kinds of information that institutional administrators recognize as valid measures of value and to develop ways of communicating the contributions that library and information services make. In order to address these two objectives, the authors conducted a five-phase project. In an earlier paper, they described the first four phases of the project and the development of the taxonomy provided in Appendix A in detail [12]. Briefly, the study entailed a review of the literature, the development of a taxonomy derived from the Balanced Scorecard Approach of LIS contributions in hospitals and AHSCs, interviews with LIS directors and institutional administrators, and a focus group of hospital administrators [13]. This paper describes the study's fifth phase, a survey used to validate the taxonomy of LIS contributions in hospitals and AHSCs, and it outlines an approach for identifying and communicating the contributions of LIS.

VALIDATING THE TAXONOMY

The taxonomy of LIS contributions lies at the heart of the CLIS approach. As seen in Appendix A, the taxonomy consists of three levels: mission concepts, organizational goals and LIS contributions. The taxonomy organizes LIS contributions around five mission-level organizational concepts. Each organizational concept is divided into subsidiary organizational goals; there are fifteen organizational goals in total. The third level of the taxonomy consists of forty-two LIS contributions associated with organizational goals. This list is meant to be representative rather than exhaustive.

To validate the taxonomy, the authors developed two Web-based questionnaires, one for LIS directors and one for institutional administrators, which focused on the fifteen organizational goals articulated in the taxonomy. As a result of extensive pilot testing involving the study's consultants and participants in the interview and focus group phases, the questionnaire instruments moved from collecting data at the level of individual measures to the broader level of assessing the potential contributions of LIS to organizational goals. Feedback from pilot testers underscored the importance of keeping the instruments brief. The researchers were concerned that the initial survey instrument, which consisted of questions about forty-two contributions, would result in a low response rate among administrators. By focusing on LIS contributions to fifteen organizational goals rather than the forty-two LIS contributions from the taxonomy, the questionnaires provided opportunities for validating the taxonomy and exploring potential differences in how institutional administrators and LIS directors perceive the potential for LIS to contribute to specific organizational goals. Also, the researchers anticipated that institutional administrators might not be familiar with specific contributions of LIS but would have clear opinions about the potential contribution of LIS to overall organizational goals. Last, the researchers considered that certain LIS contributions might be less generalizable than organizational goals.

Both questionnaires had the same structure and consisted of three sections. The first section asked both sets of respondents to rate the degree to which they believed that LIS could contribute to each of the fifteen organizational goals listed in the taxonomy. The second section asked the LIS directors to describe in their own words the most effective ways that the library's contributions to the success of the organization could be communicated; as part of this section, the administrators were asked to identify three to five performance measures that they would find most meaningful in terms of evaluating the library's contribution to the organization. The third section consisted of various demographic questions.

The sample consisted of LIS directors from sixty non-university hospitals randomly selected from the members of MLA's Hospital Libraries Section and sixty AHSCs randomly selected from the Association of American Medical College's (AAMC's) institutional members in the United States. Institutions represented by participants in earlier phases of the study were excluded from the sample. The recruitment of institutional administrators took place with the assistance of LIS directors participating in the study. LIS directors provided the names and addresses of hospital administrators. In addition, deans of medical education programs at the randomly selected academic institutions were invited to participate. After excluding administrators who indicated that they preferred not to participate in the survey, the final sample of institutional administrators included twenty-five non-university hospital administrators, fifty-one deans of medical education programs, and twenty-five academic hospital administrators.

All members of the sample received a total of three email messages inviting participation. Of the 120 LIS directors, 71 completed the questionnaire, a response rate of 59%. Of the 101 institutional administrators, 34 completed the questionnaire, a response rate of 34%. Table 1 shows the proportion of both LIS directors and institutional administrators who selected “strongly agree” or “agree” for each organizational goal to which LIS could contribute. The organizational goals that received the highest level of agreement among LIS directors as potentially supported by LIS were clinical learning, clinical care, meeting accreditation standards, and providing the resources necessary for teaching and learning. The organizational goals that received the lowest level of agreement among LIS directors as potentially supported by LIS were improving the lives of community members and increasing the organization's profitability.

Table 1 Library and information services (LIS) contributions to organizational goals: agreement among institutional administrators and LIS directors

As seen in Table 1, institutional administrators were in agreement with LIS directors about the potential contributions of LIS to two organizational goals, “Clinical Learning” and “Provide Resources and Services Necessary for Teaching and Learning.” In general, these contributions are in line with traditional perceptions of the role of LIS. The strength of agreement about the potential contribution of LIS toward the organizational goal “Promote Clinical Care” is not as strong, however, among institutional administrators as among LIS directors. The differing perspectives indicate that LIS directors may need to communicate more clearly to administrators how LIS contributes to clinical care. “Increase Profitability” is the organizational goal that received the lowest level of support among both LIS directors and institutional administrators in terms of potential LIS contributions. There are two possible explanations for these results. First, LIS directors traditionally have not identified themselves with contributing to the bottom line of the organization but rather have focused on their role of providing support to educational and research functions. Second, institutional administrators have tended to view LIS as an overhead expense rather than as a potential source of cost-savings or as a potential provider of information for use by management.

The survey validated the taxonomy of organizational goals in that at least 68% of the LIS directors agreed with all of the goals listed. Agreement among LIS directors ranged from a low of 68% to a high of 100%. Agreement among administrators was lower, from a low of 35% to a high of 97%. Based on these results, the researchers believe that all of the contributions are valid and could apply to individual settings; it is not the intent that the entire taxonomy be applied to a given setting but rather that contributions be selected. In response to the open-ended questions, participants did not identify new LIS contributions beyond those in the taxonomy. For this reason, the preliminary taxonomy presented in our earlier publication remains unchanged at this time. Additional contributions may be added based on local factors [14].

Findings from the questionnaires also point to the possible influence of setting on organizational missions and goals. Because of the relatively low response rate among administrators and the resulting possibility of self-selection bias, the following discussion is restricted to data collected from LIS directors. Of the seventy-one LIS director respondents, forty were academic LIS directors and thirty-one were hospital LIS directors.

As can be seen in Table 1, overall agreement among LIS directors in hospitals and academic settings is high. However, there are both similarities and differences in the levels of agreement about LIS contributions to organizational goals based on the responses between these two groups. In both settings, “Promote Clinical Learning” was the organizational goal that had the most agreement, and “Increase Profitability” had the least agreement. As was noted earlier, the notion of clinical learning has been a traditional goal associated with LIS in health care environments. On the other hand, contributions to the profitability of the organization have not been associated with LIS in most settings, and this appears to be consistent across organizational settings in this study.

Other differences displayed in Table 1 may also relate to organizational missions and goals associated with different settings. For example, “Provide an Organizational Learning Environment” emerged as an organizational goal with 100% agreement among hospital librarians and 93% academic LIS directors. On the other hand, “Provide Excellent Educational Programs” received 100% agreement among LIS directors in academic settings and 90% agreement from their counterparts in hospitals. The authors discuss the impact that setting may have on LIS contributions in this study's final report [15].

IDENTIFYING AND COMMUNICATING CONTRIBUTIONS

Building on the validated taxonomy of mission concepts, organizational goals, and LIS contributions, the researchers moved to the development of an approach for applying the taxonomy in specific settings. The seven-step approach described below provides the necessary tools for the identification, measurement, and communication of the contributions of LIS services, yet allows for the adaptation of the process to fit individual settings and organizations.

1. Select appropriate organizational goals

The CLIS approach is predicated on the concept that the contributions of LIS to an organization's ability to achieve its mission can be demonstrated by mapping specific LIS services to organizational goals. In order to do this effectively, it is necessary first to identify those goals that are most important for an individual organization. LIS directors should work in consultation with institutional administrators to select the goal statements they find most meaningful in their organization. This top-down approach is necessary to lay the foundation for the subsequent steps in the CLIS approach.

The validated taxonomy presents a menu of broad organizational goals in hospitals and AHSCs. However, LIS directors may need to refine these or identify new goals to reflect the priorities of a specific setting. The two organizational goals with the greatest agreement among administrators for LIS contributions, “Promote Clinical Learning” and “Provide Resources and Services Necessary for Teaching and Learning,” could logically be among the organizational goals selected. In addition, LIS directors may choose to gather data related to contributions to the organizational goal “Provide Excellent Clinical Care” because of the importance of this goal in many health care organizations. Although there was little support for LIS contributions to the organizational goal “Increase Profitability,” LIS directors might find that measuring contributions in this area could support a compelling message for institutional administrators.

It is important to select the set of goals that will be used as the starting point of the process carefully. Selecting three high-level goals that reflect the top priorities of the organization may be sufficient to demonstrate the value of LIS for an organization.

2. Link library and information services (LIS) contributions to organizational goals

Once key organizational goals have been selected, the next step is to determine the associated contributions that can be used to demonstrate the value of LIS to the organization. As shown in Appendix A, forty-two possible contributions are identified and linked to specific organizational goals; again, this set of contributions is illustrative and not exhaustive. In response to specific organizational situations, LIS directors may identify additional contributions.

As when selecting organizational goals, it is important to limit the number of contributions that are identified so that the resulting list of measures does not become unwieldy. If possible, LIS directors should discuss the contributions with institutional administrators so that the most meaningful contributions are selected and later communicated to administrators. In addition, the LIS director and staff should examine the array of LIS offered and determine which are likely to be associated with the key contributions.

For example, if contributing to the goal “Provide Excellent Clinical Care” is considered critical in a given setting, it would be necessary to measure the ways in which LIS “support informed and timely clinical decision making” and “support the development of policies and procedures relating to clinical care.” Should it become evident that demonstrating LIS contributions to the organization's bottom line is important, then LIS directors will communicate their contributions to the goal “Increase Profitability” by measuring the various ways that the provision of resources and services contribute to savings of organizational staff time, increased organizational staff productivity, reduced organizational expenditures, and lower costs of patient care.

3. Obtain data on the correspondence between LIS contributions and LIS services

After identifying the key organizational goals and possible contributions of LIS to these goals, the specific services need to be related to the corresponding contributions, since it is through the services offered that a library or information center demonstrates its value. Although LIS directors and staff may describe the intent of a service in terms of its contribution to an organizational goal, only a service's users can validate the correspondence between the service and possible contributions.

The matrix survey included in Appendix B is designed to obtain feedback from LIS users regarding the correspondence between currently provided services and each LIS contribution associated with a specific organizational goal. This instrument includes a limited number of organizational goals and LIS; it need not include all fifteen organizational goals. The selection of organizational goals and services will depend on the results of the first and second steps of the CLIS approach. Libraries in hospitals and AHSCs offer a variety of services. The selection of services for the matrix survey should be based on actual practices in a given setting.

Sampling for the matrix survey need not be exhaustive, but it should be extensive enough to provide valid results. A simple convenience sample of library or information center users may not ensure the desired level of credibility when reporting to administrators. Possible approaches for gathering data include the use of a focus group or distribution of the matrix in print or electronic form to a purposive sample of LIS users representative of the organization's primary populations.

Results of the mapping of LIS services to LIS contributions may also provide an opportunity for identifying those services that contribute to several goals and could be seen as priority services in terms of allocation of resources. In addition, the matrix may identify LIS that should be revised or adapted. That is, if an existing service is intended to help meet a specific goal, but users do not perceive it to be a contributor, the service should be evaluated to determine if it is being provided ineffectively or if the service does not contribute as intended. Understanding how services can be modified to support LIS contributions more directly may require additional data collection, perhaps through focus groups and interviews.

If a service is being revised or adapted based on the results of the matrix survey, LIS staff may consider administering the matrix survey to a sample of users on a regular basis to measure changes in ratings over time. Otherwise, the matrix survey need only be administered if LIS staff are planning to include a new contribution or service in the overall plan for data collection.

4. Select measures for services

While the mapping of current services to LIS contributions may be useful as an independent evaluation activity, the most important result of the matrix survey is to obtain an indication of which services support specific LIS contributions so that appropriate measures can be identified and communicated to administrators. The LIS director should develop a set of possible measures for each identified service that supports at least one key LIS contribution. An important criterion for inclusion of contributions in the data collection plan is that they be measurable. This may sometimes mean that significant contributions would not be included in the plan because no appropriate measures exist. Another selection criterion could be that the measures associated with a contribution are used to evaluate other intangible services within the organization.

Results of the matrix survey administered previously may inform the selection of measures as part of a logic model of LIS. Commonly used for program planning and evaluation, logic models have been applied fruitfully in the domain of public health [16]. Logic models establish the relationship between an intervention and desired results by describing the theory and assumptions underlying the provision of services. They may also guide the selection of data for monitoring and improving services. A basic logic model identifies the activities; resources or inputs; and output, outcome, and impact measures associated with an intervention or program [17].

General categories of measures that may be used for LIS are output measures, performance measures, outcome measures, and impact measures.

• Output measures can be represented by usage statistics and may include measures of use by specific user groups, intention of use, and location of use. Usage statistics are probably most meaningful when they provide insight into total usage by homogeneous groups of users. For example, clinical staff can be further subdivided into smaller groups: physicians, nurses, pharmacists, and allied health professionals. Another useful category of usage statistics is that of intention. Intentions of use are the underlying reasons behind service usage. For example, a physician may use a particular LIS to formulate a diagnosis, manage a disease, or stay informed of medical advances. A third dimension of usage is the location of the LIS user at the time of the usage. In general, LIS usage can be in-house (in the library) or remote. Remote usage could occur at the bedside, the staff or resident lounge, the waiting room, admissions area, nurses' station, physician's office, or a clinical unit, such as the intensive care unit or the emergency room. Usage data collected at the time of use has greater validity than reports of use based on users' recollection of their behavior.

• Performance measures may include turn-around time for a service as well as users' perceptions of value, satisfaction, and service quality. These measures can be used to monitor the quality of services that are associated with specific outputs, outcomes, and impacts.

• Outcome measures map usage to the outcomes of use for individual users. Outcome measures may include responses to survey questions about the results of LIS use on individuals' activities, such as clinical decision making.

• Impact measures document the results of LIS usage at the level of the organization. Examples of impact measures include graduation rates, school rankings, and an institution's aggregated clinical data. Impact measures, although specific and quantifiable, may not be isolated as directly attributable to the services provided. Logic models document this assumption of relationships between services and impacts. Table 2 shows a logic model of the relationship between reference/consultation services and the organizational goal of providing quality clinical care. Multiple logic models may be needed to describe the relationships between specific services and organizational goals.

Table 2 Library and information services logic model with possible performance measures: reference/consultation services and clinical care

5. Collect and analyze data for selected measures

Once appropriate measures have been determined, data must be gathered and presented in a meaningful way. If data indicate that a high number of physicians use library resources from the bedside, LIS directors can point to this as evidence of the LIS contribution “Support informed and timely clinical decision making,” and that this supports the organizational goal “Provide Excellent Clinical Care.” In some instances, it may be necessary to combine different measures to present a compelling case. For example, measures such as results of accreditation reviews, licensing success, test scores, and graduation rates may be associated with usage statistics. If data suggest a relationship between students' use of specific services and their performance on licensing examinations, it is possible to assert that these services contributed to the goal “Provide Excellent Educational Programs” through the contribution “Promote academic excellence.”

An ongoing data collection plan for gathering longitudinal data related to key LIS contributions for the organization is an important component of this process. The initial data collected may be used as baselines for future comparisons. Depending on the types of measures that have been selected, data collection methods may include quantitative and qualitative techniques and may be either unobtrusive or intrusive, for example, gate counts, user satisfaction surveys, focus groups, usage statistics. Depending on the individual organization, certain types of data may be more persuasive to specific administrators. As the data collection plan is developed, the LIS director will focus on those types of data that may be most meaningful for specific administrators.

6. Plan and sustain communication with administrators

Data collection is performed in the context of a larger plan for communicating with institutional administrators about LIS contributions to organizational goals. When communicating the contribution of library and information services, the value of these services needs to be described in terms familiar to administrators. As a communication strategy is developed, the LIS director selects the types of information and presentation formats that may be most appropriate for a specific supervisor. Communication needs to be ongoing and should rely on diverse methods. Most important, however, is that the measures selected for reporting provide information in context rather than as individual data points. For example, a set of current transaction figures for a given service may not be as compelling as a statement that usage has increased by a specific percentage since the previous year.

In this study, LIS directors responding to the questionnaire suggested a variety of specific methods for communicating the contributions of LIS to the larger organization. These methods can be categorized on the basis of whether they are written or oral and formal or informal:

• formal written communication, including annual or monthly reports;

• informal written communication, including email messages and brief memos;

• formal oral communication, including planned presentations; and

• informal oral communication, including unplanned conversations.

In addition, there are other nontraditional methods of communicating with administrators noted by LIS directors. For example, communication about the contributions of LIS may take place in the context of institutional activities conducted by LIS staff, such as participation on institutional committees.

Strategies for effective communication suggested by the LIS directors in responses to open-ended questions include the following:

• Reporting should be as needed—LIS directors should avoid preparing reports that will not be read.

• Both qualitative and quantitative data should be used in reports for administrators.

• When LIS directors receive positive feedback from users, they should encourage it to “percolate up.”

• LIS directors have to make themselves known by being their own public relations agents.

• It is essential to be proactive about making LIS services known.

• Whenever possible, LIS staff should participate in strategic planning processes for the organization.

Specific methods of communication offered by LIS directors include using a wide range of media, taking advantage of marketing opportunities, and involving LIS staff in the activities of the larger organization. The more general communication strategies that LIS directors offered are to be visible and proactive and to participate in the generation of grant proposals. The success of specific methods will depend upon the organizational culture and an administrator's management and communication styles. It is important for the LIS director to determine which methods of reporting are most effective for a given administrator.

7. Evaluate findings and revise selected goals, contributions, and services as necessary

The final step in the CLIS approach is to review the results in terms of whether the contributions of library and information services to the organization were effectively communicated. This may be done through a formal review with the appropriate administrators or through more informal channels. In either case, the LIS director must be prepared to revise the data collection and communication methods as necessary to support the evolving needs of the organization.

CONCLUSIONS

Directors of library and information services typically rely on data other than financial impact to communicate their value. Common measures include use statistics, data on the impact of use for individuals, and measures of users' satisfaction. The “contributions of library and information services,” or CLIS approach, described in this article may assist LIS directors and staff in identifying, measuring, and communicating the value of their services in terms of contributions to organizational goals. The foundation of the CLIS approach is a taxonomy of LIS contributions that emerged from data gathered in hospitals and AHSCs. The taxonomy identifies possible LIS contributions in the context of broad organizational goals that are common across these settings. Library directors should identify measurable LIS contributions to an organization's mission and goals in consultation with the organization's administrator. The CLIS approach depends upon establishing a relationship between specific library and information services and the goals of the larger organization. This relationship may be determined by administering a matrix survey to LIS users that relates LIS contributions and services to organizational goals. Successful implementation of the CLIS approach involves data gathering efforts to obtain the evidence needed to communicate value to administrators. Both quantitative and qualitative data may be provided as compelling evidence of LIS contributions, and these contributions should be communicated to administrators using a variety of methods. A combination of formal and informal communication channels were identified as particularly effective.

The effectiveness of individual LIS directors' actions will remain limited, however, unless there are revisions to the accreditation criteria related to libraries and information services. Specifically, the authors encourage the Medical Library Association and the Association of Academic Health Sciences Libraries to work with the Accreditation Council for Graduate Medical Education to improve the LIS-related residency program accreditation criteria. The criteria for many residency programs currently do not mention access to a library or information services. Others only specify access to a library collection and bibliographic databases without specifying the availability of training provided by a librarian [18]. Given the importance institutional administrators place on these criteria, it is critical that they reflect the broad range of library services that contribute to the success of residency programs. Such an effort will complement the work of individual LIS directors using the CLIS approach to demonstrate the contributions of LIS for specific organizations.

APPENDIX A

Taxonomy of library and information services (LIS) contributions in hospitals and academic health sciences centers

APPENDIX B

Sample matrix survey