Evaluation of a self-paced learning module to teach responsible literature searching for research

BACKGROUND

The unanticipated death of a healthy research study volunteer at a major university highlighted the importance of a comprehensive and exhaustive literature review in conducting responsible research [1]. When this unfortunate event was investigated, it became evident that many researchers who perform literature searches have little guidance about what constitutes an appropriate or sufficient search to support human subject research. The researcher in that study had completed a basic search of MEDLINE and recent texts but had not conducted an extensive search, thereby missing relevant published research [2,3]. After many of its members conducted literature searches themselves, the committee assigned to investigate this event found it was divided on what constitutes a sufficient literature search to support human subject research [4].

The Medical Library Association (MLA) recommended development of standards for literature searching to help institutional review boards assess the quality and comprehensiveness of literature searches in studies under review, and the association developed a policy statement on the role of expert searching in the health sciences [5–7]. In an article published long before this unfortunate event, Cullen and Mason decried the lack of standards for health sciences literature searching:

medical researchers are not required to account for the rigor of the literature review in the same way as they are required to account for experimental design, methodology and accurate reporting of findings in the research report which it precedes. They are trained in science but not in the bibliographical structure of their own discipline. [8]

PLANNING

To address this educational need, the Health Sciences Library System (HSLS) at the University of Pittsburgh developed “Responsible Literature Searching for Research: A Self-Paced Interactive Educational Program (RLS)” in 2003, with funding from the Association of Academic Health Sciences Libraries (AAHSL), through a cooperative agreement between the Association of American Medical Colleges (AAMC) and the Office of Research Integrity (ORI), US Department of Health and Human Services [9]. The project was proposed in response to a request for applications from ORI and AAMC for initiatives to promote the responsible conduct of research, including development of an online curriculum module.

RLS is an instructional tool aimed at teaching clinical researchers the fundamentals of effective literature searching for research practice. The program is self-paced, allowing users to work through the program with no restrictions on pace or order. The program consists of nine chapters with didactic information, practice guidelines, examples, and supplementary materials, and it includes tests and quizzes to measure knowledge gained. The RLS program objectives are to teach and describe generally accepted practices and principles associated with the biomedical literature search process. These practices and principles include appropriate use and limitations of major information resources, guidelines for responsible literature searching, HSLS resources and services available to support the literature search process, role of reference librarians in this process, and best approaches for specialized topics such as drug safety and identification of adverse events.

Because the module is based on principles applicable in most research-oriented settings, the cooperative agreement with AAMC/ORI specified that the content be freely available to other academic health sciences libraries and medical centers to adapt to their local settings and resources.

An advisory committee of experienced medical reference librarians from major institutions (Yale University, University of North Carolina at Chapel Hill, George Washington University, University of Wisconsin–Madison, and the Medical College of Georgia libraries), as well as eight HSLS reference librarians, evaluated the RLS content and test questions before deployment. The feedback received from these experienced librarians indicated they felt the program was a thorough review of searching philosophy and technique, and they approved of the depth and breadth of content and the logical flow of materials. ORI invited HSLS to demonstrate the program during the “Responsible Conduct in Research Exposition” at the annual meeting of the Society of Research Administrators International, held in Pittsburgh, October 2003 [10]. More than fifty attendees, including professional staff from ORI's Division of Education and Integrity, reviewed the draft content and provided positive feedback about the usefulness of this type of program. Ten clinical researchers from the University of Pittsburgh also reviewed the program and completed the quizzes on a pilot basis. They provided detailed feedback on content applicability, clarity, comprehensiveness, usefulness, and ease of use. RLS was formally introduced at the AAMC annual meeting and the 2005 MLA annual meeting [11,12].

IMPLEMENTATION

RLS was first made available in January 2004 through the University of Pittsburgh's web-based institution-wide responsible conduct of research (RCR) training program, “Research and Practice Fundamentals (RPF)” [13]. In May 2006, the RPF site was relaunched as Internet-based Studies in Education and Research, University of Pittsburgh Health Sciences eLearning Environment <https://cme.hs.pitt.edu>. This site currently supports institutional training mandated by government agencies and regulators, including Research Integrity, Human Subjects Research, Use of Laboratory Animals in Research & Education, Conflict of Interest, Human Embryonic & Fetal Stem Cell Research, HIPAA, and Environmental Health and Safety. RLS was incorporated into the new system as it fit the criteria for inclusion, and including it gave it the same exposure to users as the other modules in RCR. While not mandated, completing the RLS module is highly recommended for individuals who are involved in human subjects research at the University of Pittsburgh and the University of Pittsburgh Medical Center (UPMC). Anyone can view the module's content by registering and creating an account [14].

EVALUATION

Since implementation in 2004, 1,653 unique users have successfully completed the RLS module and received certification by scoring 80% or higher on its test or quizzes. Between implementation of the module in 2004 and its move to another system in 2006, 478 users completed the module. Between May 2006 and February 2009, 1,175 users completed the program. As with other modules in the RCR, users are required to complete an evaluation at the end of the module before they receive certification. When the module was moved in 2006, the evaluation form was significantly changed, and thus the data between the 2 evaluation periods cannot be compared. The results reported here are for the 1,175 users completing the module between May 2006 and February 2009.

The majority of the users (1,055) were affiliated with the University of Pittsburgh or were UPMC users, and 80 were from other institutions. Users had the option to obtain certification by taking the module test without reviewing the module's content. Of the 1,175 users, 925 (79%) users attempted the module test without reviewing the module's content. They either felt that they already had enough knowledge to pass the test, or they decided to take a chance on passing. Of those, 300 (25%) were certified through this “test out” feature.

The evaluations at the end of the module included 7 items designed to assess users' perceived outcomes as a result of completing the module. These questions were modeled after those used in the other RCR components. The responses from users were very positive (Table 1). Approximately 80% of users responded that the degree to which they expected to change the way they conducted their professional responsibilities as a result of completing the module was moderate, high, or very high. The same was true about the way users thought about their professional responsibilities. Eighty percent responded that reviewing the module moderately, highly, or very highly changed how they thought about their professional responsibilities. Approximately 90% of users responded that the module moderately, highly, or very highly enhanced their knowledge of the subject area, and 92% responded that the content was relevant to their professional responsibilities. Ninety-two percent responded with “moderate,” “high,” or “very high” that the objectives of the module had been achieved. Only 10% responded that there was high or very high evidence of commercial bias in the presented materials.

Table 1.

Responses of users who completed the evaluation questionnaire

The 7th item in the evaluation, not included in Table 1, was an open-ended question asking users if they could make any changes to the module what would they be? Only 15% (174) of the users completing the evaluations answered this question. Of those, 57% (99) recommended no changes to the module. The other comments included shortening the module, including fewer questions, including more readings, and adding video or color.

Ten health sciences libraries requested RLS content to adapt to their local settings. Only two, Weill Cornell Medical Library and the Greenblatt Library of the Medical College of Georgia, have completed the adaptation at the time of this writing and made it available to their users. The module design made it easy to adapt to other institutional settings. The RLS content is given to requesting institutions in Microsoft Word files, with the quiz questions and answers in a Microsoft Excel file. The only change required is to match the resources and names to that of the individual institutions.

In April 2006, RLS was accepted for inclusion in the AAMC Providing Online Resources to Advance Learning in Medical Education (MedEdPORTAL). The MedEdPORTAL is a web-based resource designed to serve as a prestigious publishing venue, through which faculty and medical schools can disseminate peer-reviewed educational materials [15]. Submissions go through peer review using a standardized instrument developed by the AAMC Peer Review Task Force. The instrument evaluates numerous educational dimensions of the submitted resource, based on accepted standards of educational scholarship.

CONCLUSION

RLS was developed to teach clinical researchers the fundamentals of effective literature searching for research practice. The evaluations from users clearly show that those who completed the module felt it enhanced their knowledge of the subject area. They also indicated the content was relevant to their professional responsibilities and that they expected to change the way they thought about and conducted these responsibilities as a result of their gained knowledge.

One limitation to the results should be noted. Participants were required to complete the evaluation to obtain certification. This mandatory evaluation system was employed because it matched the evaluations in the other modules in RCR and would be familiar to users of this system. There are limitations in using mandatory evaluations as they are prone to bias in reporting by the response group [16,17].

Because the module was written with a broad-based approach to literature searching, minor content revisions occur yearly. These revisions are tracked by the author throughout the year, requiring a small amount of effort in updating the RLS content.

The positive feedback from users of the RLS module suggests that this type of instructional tool can serve as a model for similar educational programs at HSLS and in other institutions. Future research should continue to identify opportunities to measure the impact of web-based learning formats for instruction.