The management of health library outreach services: evaluation and reflection on lessons learned on the VIVOS project

Abstract

Purpose: The aim of the VIVOS project was to develop and evaluate methodologies, i.e., sets of methods, for determining the value and impact of “virtual outreach” information services in the health sector in the UK.

Methods: Five different projects were recruited initially, with another two added later. Methods were largely qualitative, with over 130 interviews conducted among health professionals, complemented by postal questionnaire surveys.

Results: Identified factors that affect the successful roll-out and continued development of the projects included the need for help-desk type services to provide sustained support for new users to the services.

Conclusions: Follow-up of the projects eighteen months after the end of the VIVOS project revealed that the long-term impacts for the participating library managers included the benefits of using evidence on service outcomes, enhanced recognition locally, and greater confidence in evaluation.

INTRODUCTION

VIVOS (Value and Impact of Virtual Outreach Services) was a one-year project running from February 2000 to the end of January 2001. It was conducted by the Department of Information Studies, the University of Wales Aberystwyth and received funding from Re:source—the Council for Museums Archives and Libraries, England. The research team collaborated with information professionals working on a variety of outreach projects in various settings (rural, urban, inner city). A largely qualitative approach was taken to determine the benefits to health professionals of using each service and the way use of such services might be encouraged.

This paper discusses the methods used, then presents and discusses the main themes from the findings for marketing and management of such services, with reference to relevant management theories. A brief overview of progress since the formal completion of the VIVOS project indicates how evidence from the project was used to develop services and skills among staff.

Aims and objectives.

The aim of the VIVOS project was to develop and evaluate methodologies, or sets of methods, for determining the value and impact of virtual outreach information services in the health sector. The findings were intended to inform guidelines for project management of these and similar services—a set of such guidelines is currently in preparation. The objectives were

• to extend and refine existing value and impact methodologies.

• to assess the usefulness of multiple methods of evaluation (quantitative and qualitative).

• to evaluate the effectiveness of various training approaches.

• to develop guidelines on the methods for management and evaluation of virtual outreach services.

• to develop research and evaluation skills among information practitioners through active involvement in the project.

LITERATURE REVIEW

The development of networked information services has meant that groups of health professionals and health consumers who have previously found access to library services difficult, for reasons of time or geographical constraints, now have a wide variety of information services available. Growth in provision of services to dispersed health professionals, or to those working in remote rural areas, has been rapid. However, easier access to information does not mean that users will make use of services provided, and a review [1] of the information needs of rural health professionals suggests that health librarians need to make sustained efforts to convince this group of the benefits of using information services, thereby changing their information-seeking behavior. Experience of projects in a rural setting suggests that users value the services provided, but that training must be an important part of such projects [2–5]. Indications from a follow-up outreach project [6] suggest that services need to be tailored to meet the needs of particular groups, and that “readiness” for outreach was affected by greater awareness and experience with computers in general.

In the UK and in Australia, there is a growing trend towards linking information service and evidence-based medicine initiatives. Examples include the WISDOM project [7] in one English region, the National electronic Library for Health (UK) [8], and in Australia, the Clinical Information Access Project (CIAP) project [9]. Government policy in the UK has stressed the importance of bridging the health and social-care interface [10], and there are initiatives to make social care evidence-based [11]. Evaluation of the impact of virtual outreach services is complicated by rapid changes in health and social policies, the need to publicize services and educate potential users, and the difficulty of marketing to groups whose needs have previously been unmet. Subgroups within that population may have particular needs that are unknown to the service providers.

METHODOLOGY

Focusing the study.

The VIVOS project drew together several different outreach projects that had been set up by health libraries in acute and community trust settings within the UK National Health Service. Five sites were recruited originally, but at a later date two further sites offered to make a contribution to the study.

Since the VIVOS definition of “Virtual Outreach Services” was broad enough to encompass any type of service that enabled health care professionals to access information without visiting a physical library building, the range of services studied was quite diverse. Furthermore, given the timeframe of the project (one year to evaluate seven services), it was necessary to focus on a specific aspect at each site in order to set reasonable goals.

One of the key objectives of the study was to encourage the development of research skills among library staff by actively involving them in the research process. Meetings and a workshop with representatives from the individual sites took place during the planning stages of the project, and decisions such as which particular aspect of a service should be evaluated and the sampling techniques to be used were taken in consultation with these representatives.

As a result of these meetings, it was decided that the focus of the individual evaluations should be as indicated in Table 1.

Table 1 Projects evaluated by Value and Impact of Virtual Outreach Services (VIVOS)

Evaluation methods.

The methods used for the VIVOS study were based on those developed for earlier impact projects, e.g., the Value project [16, 17], the Establishing the Value of Information to Nursing Continuing Education (EVINCE) project [18, 19], and reflections on assessing the value of information to health professionals [20]. A follow- up study to the Value project, Getting Information to Vocational Trainees (GIVTS) [21] made more use of semi-structured interviews, and this seemed to be productive in gaining the views of those who might be infrequent users of some information services.

The emphasis was on using a “mix” of evaluation methods (qualitative and quantitative) in order to meet the twin objectives of assessing the value of using multiple methods of evaluation and refining the approaches used in earlier projects. The following methods were chosen:

• Semi-structured interviews (over 90% of which were face-to-face, with the remainder by telephone)

• Questionnaires (both postal and email)

• Critical Incident Technique (CIT) (This technique requires respondents to think back to a significant occurrence—such as a recent information need—and helps the researcher to gain an understanding of the event from the perspective of the individual. CIT was integrated into interviews and questionnaires at some sites) [22]

• Vignettes (Vignettes were used in interviews only. This technique involves using a hypothetical scenario to ascertain an interviewee's approach to solving an information-seeking problem.) [23, 24]

• Cost study (To determine the monetary benefits—in terms of staff time saved—of remote access to databases. This was conducted at one site using data from questionnaires and was led by a health economist).

Table 2 shows the “mix” of methods used at each site (North Thames is not included in the table since the VIVOS team carried out an analysis of existing data at this site). The choice of methods was made after discussion with the site representatives and took account of what would be practical. Vignettes and CIT were not used in conjunction with each other as it was thought that they would make the interviews too long and tiring for the participants. A key factor in the design of the survey instruments—and, indeed, a key challenge for research staff throughout the project—was to make the survey instruments relevant to the individual services while structuring them so as to allow common themes to be identified across all sites.

Table 2 Research methods used by the VIVOS team

Qualitative data were analyzed using the NUD*IST 4 software package, with SPSS and Excel used to analyze quantitative data.

Two workshops were held for library staff involved in the project. The initial workshop provided an opportunity to share experiences and help plan the scope of work at each site. After the empirical research was completed, a workshop was held to invite feedback from library staff on the draft report and to investigate project-management issues related to the development of these projects. About eighteen months after completion of the project, the library staff were contacted again, to find out how the projects had progressed and thus help determine the outcomes of participation in the VIVOS project.

Sampling.

The guiding principle of the sampling was to minimize bias. This proved to be more easily achievable at some sites than others. The questionnaires at Leicester, West Suffolk, and Exeter were sent out following a process of stratified, randomized sampling to ensure that a representative range of views was canvassed. At Salford and Trafford all trainees who had attended training sessions were either interviewed or received questionnaires. Some members of staff who had not received training were also interviewed with a view to assessing whether the cascading process was working effectively. South Humber interviewees were randomly selected from lists of registered service users, though the sample was stratified to represent the range of job roles on the lists. The interviews in Cornwall and West Suffolk were arranged through library staff, who were encouraged to make random selections across a range of job roles. At this point it became more difficult for the researchers to ensure lack of bias, and, in fact, some West Suffolk interviewees were classed as “selected-directed,” having been chosen by library staff because of their support for the Pink Book (a directory of local information that was originally in hard copy, but has moved to the Web, and has been expanded to include features such as guidelines and protocols [Table 1]).

Table 2 gives details of the number of interviews per site and the number of questionnaires sent out. Table 3 shows the number of interviewees by job type across all sites.

Table 3 Number of interviews conducted with interviewees from different job roles

Efforts were concentrated on obtaining as large a number of interviews as possible, which meant that little time could be devoted to following up the questionnaire surveys. The lowest response obtained to the questionnaire survey was 4% (an email survey of medical students). The response rates for the postal questionnaires were 35% at West Suffolk, 39.4% at Leicester, 44% at Exeter, and 46% at Salford and Trafford.

Limitations of the methods.

The research team was able to make use of similar, though not identical, schedules and questionnaires at each site. A major problem was the logistics of arranging and conducting interviews at so many different sites, particularly when this required traveling to various locations to meet health professionals in their workplaces. Although the original sampling aimed for a representative number of service users, at some sites there were a disproportionate number of interviewees from a particular professional group. This was mainly a result of the difficulties of contacting some types of professionals.

The attempts to reach medical students through an email questionnaire provided such a poor level of return (4%) that the data could not be included in the analysis. The fairly low response rates to the postal questionnaires could be attributed to the length of the questionnaires (which were intended to complement interview data), as well as time constraints on busy professionals. At the sites where evaluation of training was the focus, some interviewees had received training just prior to the survey work, while others had been trained for some time but had had little opportunity to put their skills into practice because of delays in the implementation of the networking. This limited some aspects of the evaluation of training.

MANAGEMENT AND EVALUATION OF VIRTUAL OUTREACH SERVICES

From the discussions at the post-project workshop and from the results of the individual site evaluations, several factors that could affect the sustainability of such projects were identified. Several of these factors reflect elements of what many management theorists term the “truth, trust, love, and collaboration” model of organizational change [25]. This model, or more correctly, collection of models, stresses the importance of purpose, establishing trust and open communication, leadership and responsibility, and the need to maintain momentum and morale.

Having a clear purpose.

Unsurprisingly, one of the critical success factors identified at the workshop was the need for a clear goal. Although flexibility of approach is necessary to cope with the unpredictable nature of projects that require participation from a variety of stakeholders, there must be an identifiable and attainable long-term “superordinate” goal [26] or “vision.” This vision has to be communicated to the users. That this had been achieved was apparent in the high motivation that was noted across the spectrum of job types and working environments, in the enthusiasm to attend training courses, and in the fact that several interviewees were keen to develop their skills outside working hours, as indicated by the following comments:

I am very interested in auditing the work we do. I am also keen to base our practice on good clinical evidence.—Nurse

And for me it was such a great achievement, to roll all that together just seemed like something that I would never ever, ever be able to do …Because I am a doer, I am not a writer or a thinker, I am a hands on person really, so it was a great achievement really.—Nurse

Integration with the local environment: building trust.

A recurrent theme when comparing implementation strategies was the amount of effort required to tailor services to the local working environments. For example, at one site, training was carried out in the workplace, and the content of each session pitched to the participants' level of competence and experience. Hardcopy evidence-based bulletins in South Humber featured “hot topics” with strong local relevance.

Virtual outreach projects cannot exist in isolation, and workshop participants noted the benefits of a supportive institutional culture that encourages staff development and recognizes that the library has a role in a “lifelong learning agenda”—that is, the library contributes to “education” rather than simply supplying training courses. Full engagement of stakeholders is essential, and responsibilities must be clear, with service-level agreements set up, if appropriate. At the post-project workshop, the library teams agreed that good communication with stakeholders, for example, IT departments, was a key factor in the success of their projects, although it had not always been easy to achieve.

Motivation of library staff members themselves should also be nurtured with involvement in the change management process and consistent support at the home base. This is equally appropriate for staff members not directly involved in the project but who may have to take on extra responsibilities. Their awareness and training needs should be identified and met to prevent enthusiasm from waning.

Raising awareness and sustaining motivation.

Interviews with users revealed widespread appreciation of the skills and support provided by librarians, but it was felt that their services could be more widely advertised. One of the most effective methods of promoting services appeared to be the recommendation of satisfied users who went back to units and enthused about library services to colleagues, confirming the wisdom of the approach used in the Clinical Information Access Project (NSW, Australia) where clinical representatives are used to “spread the message” [27]. For example:

I found that a lot of colleagues don't actually know much about [the] database. So what I did was, having been in touch with the library and having used it myself, I have in turn …spoken to my senior manager …and she has in turn initiated group sessions …of going to the library and people being trained up to a standard.—Nurse

I realized that there was a resource there that I could use, and I have used it again and again since then. But …I can't go around other surgeries and spread the gospel …people need to be told somehow that there is a resource there, it is a brilliant resource and it must cost the health authority thousands and thousands to keep it up-to-date, but do people use it? So I, I think they [librarians] need to sell themselves a bit more, do a bit of a PR job themselves.—Community-based manager

“Spreading the gospel” was behaviour typical of a group of interviewees termed “enthusiasts” by the researchers. During the qualitative analysis it became clear that interviewees tended to fall into one of three categories relating to their attitudes to and acceptance of new technologies and services. The researchers named the three categories “enthusiasts,” “discoverers,” and “pre-germinators.”

The characteristics of the three groups were noted, but further analysis would be required to establish the proportional and demographic representation of each group within the sample. Even with further analysis it would be difficult to draw conclusions about proportions and demographics because the categories emerged from qualitative interview data that are not easily quantifiable. Furthermore, it is likely that the results would be skewed by the fact that many of the participants were self-selecting in that they had chosen to attend training sessions or had registered for services. As indicated in later discussion of the cost study, the enthusiasts are probably a much smaller group than the discoverers or the pre-germinators.

Studies of the diffusion of innovations have classically divided people into categories of innovators, early adopters, early majority, late majority, and laggards [28]. On later reflection, the Rogers model could have been applied to the VIVOS data, but since the researchers had already constructed the three categories of enthusiasts, discovers, and pre-germinators it was decided that these terms should be retained. As the constructs had emerged from the data, it would not have been easy to break them down further into the five categories suggested by Rogers. It was also felt that “pre-germinators” had fewer negative connotations than “laggards,” and it seemed more appropriate since poor awareness of library resources could be partly because of lack of effective promotion by library staff or lack of support from management.

The enthusiasts displayed high levels of motivation and awareness of services. Some suggested that libraries should increase awareness of facilities amongst other practitioners while others have actively promoted, or would promote, services within their work environment (as in the examples quoted above). This group combines elements of the change agent as well as the early adopter.

The discoverers indicated that exposure to the outreach services had significantly widened their horizons both in terms of actual resources available and the expansion of personal networks. Innovation diffusion theorists have identified five critical characteristics for effecting change: relative advantage, compatibility, complexity, trialability, and observability. For the discoverers, the relative advantages of using the service seemed clear, and they were learning how to fit the greater access to information into their routines. For example:

In fact, I was absolutely floored that there was so much information that I could actually then access onto the computer, you know, which at times is brilliant.—Clinical practitioner

One of the things I have gained from the course I suppose that wasn't the main agenda is that I have met the librarians and I know who they are. And, for example, with me having students I have …been able to ring [the librarian] …and say “Is it OK if this student comes up and introduce them so that they can use the system?” …and I didn't know anybody before we did the training and so I might not have felt quite so …confident in doing that.—Nurse

The “pre-germinators” were more akin to the laggards of the diffusion of innovation theory and were unaware of available services, or in some cases they did not realize they were entitled to access them. This is not necessarily because they have no information needs, and it is possible that wider promotion of information services by library staff could stimulate growth in their awareness and motivation. An unanticipated outcome of the VIVOS study was that in some cases it helped awaken interest, particularly as this group seemed to expect that others should take responsibility for making them aware of the benefits and that reminders would be necessary:

No, I haven't personally looked at it, but if my admin manager was here she'd probably have hit me by now and said “we have got the Pink Book, it's by the front desk and we use it all the time, and it's only out of ignorance that you don't.”—Nurse

Comparing the findings of the qualitative evidence with the complementary quantitative cost study of database access conducted for one outreach site suggests that the enthusiasts are very visible service users, and that they constitute the group for whom usage of the networked databases substituted in whole or part for usual use of the library. Desktop access means time savings (and cost savings in staff time) if these users are taking less time to retrieve information they need. From the cost study, the discoverers were the new users. This group consists of those who previously would not have used the alternative library facilities to find information and have been attracted by the relative ease of information use the system provides. What the cost study demonstrated was a highly skewed distribution of usage. While the mean saving associated with using networked databases was relatively large, that was because of the small number of respondents for whom there were very large cost savings (the enthusiasts were often senior staff). The median value was therefore much lower, by a factor of around 26 times less. The discoverers are learning to use the system, their usage may be infrequent, and, from the economic perspective, their usage may incur costs to the health service since they are substituting time spent in health care for learning to search effectively, although that investment in time should reap benefits later.

However high the level of motivation among users and library staff at the start of a project, maintaining it can be difficult. This is especially true when services are faced with challenges such as the delayed rollout of IT equipment, lack of dedicated time for health professionals to carry out research activities, or a perception that such activities are only appropriate for structured continuing professional development. Infrequent use of services means that skills gained at a training session are easily forgotten and disillusionment can set in.

I think you have got to negotiate time, we don't actually have time built in, and I know a few of us have suggested that professionally there should be time for reading of whatever sort, and doing this sort of thing.—Clinical practitioner

Continued support for users.

Another common theme across the projects was the need for continued support for the users. Post-training catch-up sessions were introduced in one outreach service in response to the perceived need to respond to queries that arose when users started to practice on their own. At other sites, a 24-hour help-desk service was being considered to solve this need for ongoing support. Another approach was that of cascade training, with those trained expected to cascade skills to their colleagues. Observations suggested that barriers to successful cascading of training included low levels of confidence, lack of time, and logistical problems relating to a lack of IT equipment. The lessons learned from this indicate that if trainers do intend for a cascading process to take place they should monitor it and perhaps actively “prime the pump” in the initial stage.

POST-PROJECT REFLECTIONS

The VIVOS evaluation contributed to the review process of the participating projects, and librarians at the second workshop commented that added value came from involvement in a wider study that raised the profile, and hence awareness, of their projects at local and regional levels. A later set of knowledge-management projects has taken an action learning approach to help support their evaluation [29].

After completion of VIVOS, librarians involved were invited to comment on the long-term outcomes of participation in the evaluation. Eighteen months later, the virtual outreach services have evolved in different ways, and, while it is impossible to attribute some outcomes directly to working within VIVOS, some benefits seem clear (Table 4).

Table 4 Post-VIVOS progress

Organizational changes in the health service since the formal completion of the VIVOS project mean that some of the library service project managers are no longer in the same post, and the focus of the projects has evolved to accommodate new policy requirements. Nevertheless, one of the main benefits was the ability to provide evidence, based on the evaluation findings of more than one project, to senior managers to support funding for such service developments. The methodology has been adapted and applied in later evaluations of database outreach projects, and the cost- analysis approach was used by the researchers in an evaluation of the National electronic Library for Health. Several of the librarians now have greater responsibilities for their own research and evaluation projects.

CONCLUSIONS

Several of the objectives for the VIVOS study focused on assessing the appropriateness of multiple methods of evaluation for value-and-impact studies of this type and on testing methods used in previous research (e.g., vignettes and CIT) in different settings. It was hoped that the resulting set of methods (all survey instruments were included in the project report) could be adapted for use by other virtual outreach projects wanting to evaluate their service. Using a mix of methods proved to be valuable in terms of triangulation, and the interviews with users were particularly enlightening, although they required much time and effort to arrange and analyze. Evaluation on the scale of the VIVOS project might not be feasible for all outreach services, but interviewing some “discoverers” and the “pre-germinators” does indicate how the services need to be developed and promoted to make their usage more effective. Random sampling also avoids the problems of bias, or being swayed by the views of the few enthusiasts.

The experience of the VIVOS project has led to the following conclusions about working with the methodologies:

• Personal interviews provided insights that would have been hard to obtain from questionnaires alone.

• Use of the Critical Incident Technique or vignettes allowed the users to tell their own stories about information seeking, rather than having a library perspective imposed on their use of services.

• If using both interviews and questionnaires, the survey instruments should be designed with care to ensure that they complement each other.

• The very low response to the email survey has led the authors to be wary of relying wholly on this method in future research.

• The research team used qualitative data analysis software for the interview transcripts but simple thematic coding could be carried out manually to save time.

• A rigorous approach to sampling of research subjects is advisable—in the VIVOS project this was more easily achieved when the researchers carried out the sampling process, although help from the site library staff was much appreciated and almost certainly secured greater numbers of interviewees.

• Although the evaluation was initially intended to focus on a narrow range of topics, at some sites, having the flexibility to take a wider perspective in interviews proved valuable in helping to relate to participants.

The findings suggest that it is necessary to keep the new users motivated, and that their involvement needs to be encouraged in some way or another, particularly if these newly trained users are meant to cascade their skills to colleagues. Library staff may need to ensure that they take advantage of suitable workplace opportunities for maintaining the education and training momentum, with projects such as the development of clinical guidelines and integrated care pathways. The cost study confirmed the dangers of being swayed by any massive increase in service usage that is almost entirely due to a small number of enthusiasts. The enthusiasts can potentially be asked to serve as project champions and play a vital role in the continued evolution of such projects.

The outreach project staff appreciated closer involvement in the operation of a research project and several of those involved have consolidated their skills in new projects. To involve practitioners in research in this way may add to the time involved for management of the research project, and may mean that the quantity of data collected is less than it might have been for a wholly independent evaluation. On the other hand, there are benefits in terms of the quality of the data collected. The main aim should be the implementation of evidence-based practice in health librarianship, and the VIVOS project provided one perspective on how this might be achieved.