Time to rethink the role of the library in educating doctors: driving information literacy in the clinical environment

Abstract

Question:

Can information literacy (IL) be embedded into the curriculum and clinical environment to facilitate patient care and lifelong learning?

Setting:

The Australian School of Advanced Medicine (ASAM) provides competence-based programs incorporating patient-centred care and lifelong learning. ASAM librarians use outcomes-based educational theory to embed and assess IL into ASAM's educational and clinical environments.

Methods:

A competence-based IL program was developed where learning outcomes were linked to current patients and assessed with checklists. Weekly case presentations included clinicians' literature search strategies, results, and conclusions. Librarians provided support to clinicians' literature searches and assessed their presentations using a checklist.

Main Results:

Outcome data showed clinicians' searching skills improved over time; however, advanced MEDLINE searching remained challenging for some. Recommendations are provided.

Conclusion:

IL learning that takes place in context using measurable outcomes is more meaningful, is enduring, and likely contributes to patient care. Competence-based assessment drives learning in this environment.

INTRODUCTION

Educators are responding to the complexities of today's medical knowledge by developing educational programs based on current learning theories, such as enactivism, where learning takes place within teams that are actively engaged in clinical environments [1, 2]. Incorporating enactivist learning theory into the care of the hospital patient makes all in the medical team (both the expert and the student) partners in their personal commitment to learning. In this model, all learning activities revolve around patient care. This learning must include access to evidence-based medical literature to inform decisions, thus making information literacy (IL) a central activity of the team responsible for the care of patients. Assessment in such environments is becoming competence-based, where proven competence, rather than years-in-training determines progression [3, 4]. Learning becomes individualized and more meaningful as trainees proceed at their own rate, taking less or more time as required to become competent. The assessment of mastery of competence in a clinical environment, including competence in literature searching, correlates with effective, safe, and competent care of the patient and the contribution that each team member makes to this outcome. The Royal Australasian College of Surgeons (RACS) training programs have adapted the Royal College of Physicians and Surgeons of Canada (CanMEDS) model that defines key roles, competencies, and learning outcomes for surgeons, moving away from the previous time-based programs [5].

The Australian School of Advanced Medicine (ASAM), Macquarie University, Sydney, provides postgraduate training using principles of competence-based assessment, team-based patient care, and lifelong learning. In this environment, assessment drives learning and becomes a powerful motivator for independent, lifelong learning. Clinicians entering ASAM programs are qualified medical practitioners (not interns or residents) who are returning to study to undertake specialty or subspecialty training at master's or doctoral degree levels, while participating in the active care of patients. However, many clinicians entering ASAM programs are not proficient in conducting effective literature searches. To effectively practice evidence-based medicine and become lifelong learners, these clinicians must learn how to independently search for and manage the literature in their patient-centred context. Without these skills, they will be unable to develop their core roles as “medical experts,” as defined by CanMEDS, RACS, and other educational bodies.

The Accreditation Council for Graduate Medical Education, CanMEDS, and RACS all highlight the importance of competence in efficiently retrieving and critically appraising the literature for an appropriately formulated question [5–7]. However, many studies claim there is little evidence to support long-term retention of IL instruction. Cullen and colleagues demonstrate that clinicians' retention of literature searching and critical appraisal skills taught at the undergraduate level are not retained after graduation, unless they are practiced in postgraduate programs [8]. Garg and Turtle's review of the effects of IL training on health professionals' literature searching skills reports limited evidence of short-term benefits and recommends studies that objectively measure long-term effectiveness and learning transfer to clinical contexts [9]. Many studies are based on IL instruction taught in classroom settings that are not meaningfully linked to the clinical environment. Bradley and colleagues conducted a study showing how IL learning that takes place with librarian support in real-time clinical environments facilitates retention of searching skills up to six months later [10]. IL programs often lack objective means of measuring learning outcomes, relying instead on more subjective measures such as surveys. Brettle's review stresses the importance of providing search skills development in clinical settings using valid and reliable measures of learning outcomes [11]. She reports librarians frequently experience difficulty in embedding IL into clinical settings and objectively measuring learning [11]. Rana and colleagues state that being able to search the literature effectively is a prerequisite for practising evidence-based medicine; however, one of the issues facing librarians is the lack of validated assessment tools for database search skills [12].

SETTING

ASAM is co-located on campus with Macquarie University Hospital. Both were founded on the Mayo Clinic principles of providing progressive medical education and team-based patient care [13]. ASAM's curriculum incorporates enactivist principles, where collaborative learning takes place in clinical contexts such as operating rooms, clinics, wards, and clinical meetings, and there are no traditional lectures, tutorials, or examinations. The emphasis is on learning rather than teaching, and students and teachers are encouraged to engage in self-directed, lifelong learning and research. In this context, students are known as “scholars” and teachers are “advanced scholars.” Scholars and advanced scholars work and learn together for improved educational, research, and patient outcomes. Scholar numbers are limited to facilitate a collaborative, personalised environment in which contextual learning occurs.

Macquarie University Library is located at the centre of the university campus and provides services and resources to more than 30,000 students and approximately 2,000 academic and professional staff across 4 faculties. A team of 12 full-time equivalent liaison librarians support postgraduate students and academics across the university as well as carry out reference duties for all clients at the library's service points. Two liaison librarians provide 20 hours of services to ASAM each week by participating in meetings, ward rounds, and other clinical settings where learning takes place. Thus, the opportunity arose to create meaningful IL learning activities and to find effective ways of measuring learning outcomes that are consistent with ASAM's philosophy. This case study uncovers an emerging role for librarians as members of the educational and health care teams, where they are well positioned to embed and assess IL.

PROGRAM CHARACTERISTICS

During the early stages of ASAM's neurosurgery program, advanced scholars set up weekly journal clubs, research, and case meetings where scholars could present the results of their literature searches relevant to the care of the patients currently managed at Macquarie University Hospital. However, scholars varied in their literature searching skills, and many could not conduct effective MEDLINE searches using either OvidSP or PubMed interfaces. The librarians initially ran small classes in searching databases; however, while some scholars saw this as helpful, most returned to their pre-enrollment methods of literature retrieval. To overcome the limitations of small group instruction, the librarians also provided one-to-one instruction as required. This individualized approach was useful for creating learning experiences on an “as needs” basis but did not reach those users who “don't know what they do not know.” Neither small group nor individualized instructional methods measured learning outcomes, and while informal feedback was mostly positive, scholars' literature searching for case presentations often lacked searchable questions and effective search strategies.

The librarians also created an online subject guide where library services and resources could be readily accessed from a single website. A number of user guides and tutorials were included to provide literature searching support as and when required. Yet website usage statistics remained low, and clinicians continued to struggle when presenting their searches at case meetings.

At that point, it became evident that a new approach to IL was needed that aligned with ASAM's educational and clinical principles. The questions searched needed to have immediate relevance to patient care, the findings had to be of importance to the team, judgment as to the quality of the activity needed to be made by advanced scholars with the assistance of expert librarians, and scholars needed to reflect upon their performance. Senior neurosurgeons and librarians drafted a new weekly case presentation meeting format, which was renamed the “Evidence-Based Surgery” (EBS) meeting. These meetings would enable scholars to develop their searching skills, while finding the best available evidence to improve the care of their patients.

PROGRAM AIM

How can literature searching, learning, and assessment be embedded into the curriculum and clinical environment to facilitate lifelong learning and patient-centered care?

PROGRAM OBJECTIVES

The program's objectives are:

• to embed IL activities into ASAM's curriculum and clinical environment and to measure their efficacy through mastery of specified learning outcomes and

• to evaluate the IL pilot as a basis for expansion into other ASAM programs and clinical settings.

A one-year IL pilot with ASAM's neurosurgery programs was undertaken to explore these objectives. The librarians and advanced scholars broke down the generic ASAM learning outcome for literature searching (i.e., “Conduct a literature review on aspects of the case of a patient with the relevant condition/undergoing the relevant procedure”) into a series of smaller components called “performance elements.” They were based on the performance assessment criteria of the Association of College and Research Libraries standards [14] and the CanMEDS competencies [7]. Scholars must demonstrate the ability to meet each performance element of the learning outcome for conducting a literature search and review. The performance elements are: “Determines nature and extent of needed information,” “Develops search strategies,” “Evaluates quality of retrieved information,” “Broadens search,” “Manages resources” and “Reviews search processes.” Scholars must demonstrate mastery of the IL learning outcome and its performance elements, along with all other program learning outcomes, to become eligible for their academic award of master's degree.

At the commencement of each program, scholars are provided with performance assessment forms (PAFs), which are graded checklists incorporating learning outcomes and their performance elements for the entire program. Each PAF enables scholars to understand which tasks are being assessed and how. Refer to the online only appendix for the IL PAF. The IL learning outcome and performance elements assess scholars' competence in searching and evaluating the literature relevant to their patient cases. Learning is individualized as each scholar takes the necessary time to become competent in the required learning outcome. The learning emphasis is on the process of literature searching to find the best evidence for answering clinical questions. It is not about appraising selected or available studies, as in journal clubs, because that activity does not incorporate searching for the best evidence for improving patient care.

Development of literature searching skills takes place through scholar-initiated meetings with a librarian before an EBS presentation. These meetings are mandatory until each scholar is deemed competent in literature searching. If face-to-face meetings are not possible, email and telephone contact are organized. The scholar and librarian work through the patient problem and develop a searchable question—often using the patient/problem, intervention, comparison, outcome (PICO) format—as well as search strategies. The practice of evidence-based surgery can be problematic; for example, randomized controlled trials are difficult to conduct in many surgical areas due to ethical and technical issues [15, 16]. Scholars quickly become aware of these limitations and are encouraged to look for the best available levels of evidence when searching the surgical literature for patient care information.

Several features unique to ASAM scholars' level of literature searching set this program apart from many other assessable IL activities, including Rana's validated search assessment tool [12]. Neurosurgery scholars are undertaking specialty training where specific and complex literature searches are required that cannot be easily evaluated using standard assessment tools. For example, ASAM trainees frequently need to combine Medical Subject Headings (MeSH) with keywords due to the limited availability of MeSH terms for neurosurgery concepts; however, this skill is omitted by Rana's assessment tool. The use of limits is a critical search element in Rana's tool, but ASAM trainees are not always required to use the MEDLINE search limits as this can further reduce an already small search result [12].

Each week, the EBS meeting is scheduled alongside the other neurosurgical meetings (e.g., neuroradiology, neuropathology). During the EBS meeting, a scholar who has previously been assigned a patient case during clinics, rounds, or meetings presents this case to the group using a PowerPoint template. Each presentation includes the patient problem, history, relevant radiology and pathology, and a literature search leading to a clinical conclusion based on the best available evidence. The EBS meetings provide scholars with opportunities to not only develop skills in searching for and assessing literature for a patient problem, but to also hone their presentation skills. The meeting format provides opportunities for valuable formative assessment and feedback from colleagues and librarians. These exchanges also provide learning experiences for other meeting attendees.

The librarian, in consultation with the supervising advanced scholar, grades the scholars on each performance element by completing the PAF. For each performance element—for example, “Reviews search strategy and implements revised search”—one of the following rankings may be assigned: “Not assessable in this case,” “Not assessed in this case,” “Needs more practice,” and “Competent.” If the scholars demonstrate satisfactory performance on each of the performance elements, they are deemed competent and are signed off on the learning outcome. If scholars are not yet competent, they will practice and present another search for assessment. Global performance rankings for overall assessment include: “Not assessed,” “Not yet satisfactory,” “Borderzone,” and “Satisfactory.” In keeping with the team approach to learning and patient care and the program's focus on competence-based training, traditional ranked gradings are not provided, and scholars are simply assessed as being competent or not yet competent.

PROGRAM EVALUATION

Evaluation of any educational program has, at its core, whether or not the curriculum delivers its learning objectives and how it can be improved. In this case, the validity of the IL learning activities in becoming a competent neurosurgeon is reflected in formative evaluation of patient outcomes. The number of times that the outcomes of IL activities are employed in the decision making at the bedside is a very important marker of the success of the IL program. Metrics are being developed to objectively ascertain the degree of success of this IL program within the broader surgical curriculum.

The pilot was also evaluated through scholar and advanced scholar feedback, survey results, and examination of the PAFs, where the number of searches assessed as “Competent” or “Not yet competent” could be evaluated. During the pilot, 23 presentations from 8 scholars were assessed using PAF checklists. Performance element data for each scholar were collated in an online spreadsheet. These data indicated that, while several scholars showed improvement over time in developing searchable questions using PICO formats, the MEDLINE “Map to Subject Heading” feature, and cited reference searching, others found aspects of MEDLINE searching difficult to master. These aspects included using MeSH with keywords, understanding the “Explode” and “Focus” functions, and reviewing initial searches. Scholars also required more practice in reflecting on the value of the search process to their clinical decision making. Figure 1 shows a graph of assessment rankings of key performance elements for all presentations, demonstrating an overall improvement in searching skills from the first 12 presentations to the subsequent 11 presentations. The number of performance elements ranked “competent” increased, and the number of performance elements ranked “needs more practice” decreased. For example, during the first 6 months of the EBS meetings, or 12 presentations (Figure 1, 1 EBS), only 42% of scholars were rated competent in carrying out the advanced MEDLINE search strategies listed on the PAF. This figure rose to 73% during the subsequent presentations (Figure 1, 2nd EBS).

Figure 1.

 1 Proportion of performance elements assessed as competent in the first 12 “Evidence-Based Surgery” (EBS) meeting presentations (1st EBS) compared to the second 11 presentations (2nd EBS), demonstrating improvement over time within the group

Most importantly, several scholars have used their EBS literature searches in published, peer-reviewed articles and in conference presentations, while others have used their literature search skills to support their research work. Observations by librarians and advanced scholars, as well as scholar feedback indicate that scholars have become more confident and proficient in presenting literature searches and reviews to their colleagues in other settings, including journal clubs, research presentations, and neurosurgery debates.

The scholars requested a repository of their EBS presentations for future use, and these have been added to their subject guide as learning and reference tools. As the EBS meetings progressed, scholars requested more assistance in appraising the literature. This led to the creation of a critical appraisal subject guide where selected resources could be readily accessed. Thus, the lifelong learning cycle continues as scholars create their own learning resources as needed.

LESSONS LEARNED

The IL learning outcome and performance elements were difficult to measure objectively, so the librarians are developing additional criteria for assessing each performance element in 2012. Table 1 gives an example of detailed assessment criteria for MEDLINE searching that will enable more accurate, reliable measures to be used by librarians and advanced scholar assessors. Librarians will provide increased support for MEDLINE searching by meeting with all new scholars before their first EBS presentations and introducing advanced MEDLINE searching skills, a performance element where many scholars experience difficulties.

Table 1.

Example of assessment criteria developed for information literacy performance assessment forms

Scholar feedback recommended that more preparation time be allocated to the EBS meetings. From 2012, all scholars will be required to meet with their advanced scholars three weeks before their presentation dates. They require this time for developing a searchable question, initiating and revising search strategies, following up on results, and preparing their presentations and meeting with librarians and advanced scholars as needed. This prolonged time will ensure that the scholars have sufficient time to develop a deep understanding of the skills of the search rather than just perform the steps of the search. Literature searching in this context becomes a different learning experience compared to “just in time” searching that also takes place in the clinical setting.

FUTURE DIRECTIONS

Librarians and advanced scholars need to invest their time and skills in preparing the program, supporting scholars, providing feedback, and evaluating outcomes. Advanced scholar support will continue to be encouraged, thereby enabling scholars to access appropriate learning opportunities and receive feedback. This is sometimes problematic as advanced scholars are time-poor surgeons who need to continually balance their clinical, research, and teaching responsibilities.

Although the PAFs are a qualitative means of measuring learning outcomes, they provide useful feedback and direction for future learning. However, the degree of objectivity should improve with the use of additional performance assessment criteria, as described in Table 1. As scholar numbers increase with the commencement of new ASAM programs, the provision of liaison librarian services to ASAM will be reviewed. There will also be a need to develop sustainable online support for increasing numbers of scholars who are undertaking programs off campus.

The EBS IL program is being modified and trials are being conducted to include other ASAM disciplines, including programs in medical education, critical appraisal, medical ethics, and musculoskeletal medicine. As several new programs will be delivered in online formats, the librarians are developing videos demonstrating literature searching to support these scholars as well as providing assistance through videoconferencing, email, and telephone contact. This IL program can be successfully introduced wherever it is possible to embed learning into clinical contexts and provide feedback and competence-based assessment in supportive environments. The goals of such a program should be adapted to each clinical and educational environment. However, in traditional educational settings with large numbers of medical students, this individualized approach to IL learning may be difficult to sustain, thereby requiring changes to learning activity delivery modes or assessable performance elements.

CONCLUSION

Evaluation of this program indicates that IL is facilitated through an integrated approach where skills are embedded, practiced, and assessed in clinical settings to support patient care and lifelong learning. Competence-based assessment in this context becomes a powerful learning motivator. Further work needs to be done to create valid, reliable measures of IL skills acquisition and to measure the impact of IL on patient care. Future studies will determine the nature and extent of IL integration into other ASAM educational and clinical programs. The professional relationship between clinicians and librarians has been greatly enhanced by the integration of these librarian services into this unique learning environment.