Instructional methods used by health sciences librarians to teach evidence-based practice (EBP): a systematic review^,,

Abstract

Background

Librarians often teach evidence-based practice (EBP) within health sciences curricula. It is not known what teaching methods are most effective.

Methods

A systematic review of the literature was conducted searching CINAHL, EMBASE, ERIC, LISTA, PubMed, Scopus, and others. Searches were completed through December 2014. No limits were applied. Hand searching of Medical Library Association annual meeting abstracts from 2009–2014 was also completed. Studies must be about EBP instruction by a librarian within undergraduate or graduate health sciences curricula and include skills assessment. Studies with no assessment, letters and comments, and veterinary education studies were excluded. Data extraction and critical appraisal were performed to determine the risk of bias of each study.

Results

Twenty-seven studies were included for analysis. Studies occurred in the United States (20), Canada (3), the United Kingdom (1), and Italy (1), with 22 in medicine and 5 in allied health. Teaching methods included lecture (20), small group or one-on-one instruction (16), computer lab practice (15), and online learning (6). Assessments were quizzes or tests, pretests and posttests, peer-review, search strategy evaluations, clinical scenario assignments, or a hybrid. Due to large variability across studies, meta-analysis was not conducted.

Discussion

Findings were weakly significant for positive change in search performance for most studies. Only one study compared teaching methods, and no one teaching method proved more effective. Future studies could conduct multisite interventions using randomized or quasi-randomized controlled trial study design and standardized assessment tools to measure outcomes.

Guyatt first introduced the phrase “evidence-based medicine” (EBM) in 1991 [1]. This five-step process became the standard for how physicians would incorporate evidence into clinical practice: (1) asking an answerable question based on an information need; (2) acquiring the best evidence to answer the question; (3) appraising the evidence; (4) applying the evidence along with one's clinical expertise and the patient's or population's preferences, values, and circumstances in practice; and (5) assessing how efficiently and effectively the previous steps were done [2]. This approach has since been utilized in many other health-related fields—including nursing, social work, and public health—and is commonly referred to by the broader descriptor of evidence-based practice (EBP) [3].

Many curricula for health professional education include EBP components to fulfill accreditation requirements and follow national recommendations, such as those from the Association of American Medical Colleges (AAMC) and Royal College of Physicians and Surgeons of Canada [4, 5]. As many EBP skills are closely tied to information literacy and research has demonstrated the effectiveness of information literacy instruction in other disciplines [6–8], librarians are natural candidates for teaching EBP [9, 10]. This systematic review aims to determine the methods employed by librarians for teaching EBP within health sciences curricula and which instructional method is most effective in promoting successful learning.

METHODS

The research methodology used for this study was a systematic review based on guidelines outlined in the “PRISMA Statement” [11]. A systematic review published by Koufogiannakis and Wiebe was used as a model for conducting a systematic review in the field of library and information science [12].

Eligibility criteria

The following criteria were used to screen and select studies about instruction to be included in the systematic review.

Inclusion criteria

• ▪Instruction must be led by a librarian.
• ▪Instruction may be in any format, including face-to-face, online, or a hybrid of both.
• ▪Instruction must be related to EBP including concepts such as population, intervention, comparison, outcomes (PICO); the evidence-based medicine cycle; searching of the literature to apply to clinical practice; or critical appraisal.
• ▪Instruction may be integrated into the curriculum or be a standalone course.
• ▪Study population must be health sciences–related at the diploma, undergraduate, or graduate level.
• ▪Instruction must include assessments of learner skills.

Exclusion criteria

• ▪Instruction was focused solely on information literacy.
• ▪Instruction was a one-shot session and/or library orientation.
• ▪Instruction was geared to a study population of practicing professionals in health sciences–related fields or in veterinary studies.
• ▪Instruction was assessed using only feedback forms or measures of learners' self-perceptions of improvement.
• ▪Letters, editorials, and comments were excluded.

Search strategy

A base search was developed and tested in PubMed using relevant search terms, both Medical Subject Headings (MeSH) and keywords, for the three key concepts: instructional methods, librarians, and EBP (Appendix A, online only). This base search was then customized for each database search to reflect database-specific subject headings. No limits were applied. Seventeen different databases and Google Scholar were searched through November 2013. An updated search of all databases was conducted in December 2014. The following databases were searched:

• ▪Campbell Library (2004–)
• ▪CINAHL bibliographic (EBSCO; 1981–)
• ▪CiNii Articles (1980–)
• ▪Cochrane Library (1992–)
• ▪Education Resource Information Center (ERIC) (EBSCO; 1966–; ProQuest; 1966–)
• ▪EMBASE Classic (Ovid; 1947–)
• ▪Embase.com (1974–)
• ▪Google Scholar (exported first 500 results)
• ▪Igaku Toshokan (1954–)
• ▪Library and Information Science Abstracts (LISA) (ProQuest; 1969–)
• ▪Library, Information Science & Technology Abstracts (LISTA) (1964–)
• ▪LILACS (Virtual Health Library; 1982–)
• ▪MEDLINE (Ovid; 1946–)
• ▪PsycINFO (Ovid; 1806–)
• ▪PubMed (National Library of Medicine; 1946–)
• ▪Scopus (1966–)
• ▪Web of Science (1900–)

In addition, Medical Library Association (MLA) annual meeting abstracts were hand searched from 2009 to 2013 for relevant studies, and the 2014 meeting abstracts were searched in December 2014. All database search results were imported into EndNote Basic. Duplicate items were identified and removed prior to study selection.

Study selection

In selecting studies to include in the systematic review, search results were divided into five sets, and each set was assigned to two members of the research team. Each pair independently reviewed their sets of results against the inclusion and exclusion criteria, using the title and abstract of each item. Each pair compared their results and removed all items where both had noted to exclude. For records that the pair did not agree, a third reviewer made the final decision. This smaller set was then examined using the full text of each article. Once again, for records that were not agreed upon by the pair, a third reviewer was used to make the final decision of which articles to include in the systematic review. The pairs also identified any duplicates that had not previously been located, and these were removed. The results from the five groups were combined to make the set of items for data extraction. This set included forty-nine items (thirty-eight from the initial searches conducted in November 2013 and eleven items from the updated searches conducted in December 2014).

Data extraction

Data extraction was performed on the forty-nine items, modeled after the method described by Koufogiannakis and Wiebe [12] (Appendix B, online only). Nineteen data elements were extracted from each study by two members of the research team and recorded in an Excel spreadsheet. Each pair independently extracted the data, compared their results, and removed all of the articles where both had noted a criterion-based reason to exclude. For any items that were not agreed upon by the pair, a third reviewer made the final decision.

Critical appraisal

Critical appraisal was performed concurrently with data extraction on the same set of forty-nine articles to determine the quality, validity, and applicability of each study. The nine-item Glasgow checklist was used for the appraisal [13]. A different pair of team members performed the critical appraisal on the articles.

RESULTS

Study selection

A total of 49,795 bibliographic records were identified through database searching (45,157 in November 2013, plus another 4,638 in December 2014). An additional 456 records were identified through the MLA annual meeting abstracts (448 in November 2013 and an additional 8 in December 2014). After removing duplicates, 30,043 remained for screening. In selecting studies to include in the systematic review, the research team was divided into pairs, and each pair independently reviewed approximately 5,000 citations using only the title and abstract. Following title and abstract review, 29,406 studies were excluded leaving 637 studies for full-text review. After full-text review, 49 studies were selected for data extraction and critical appraisal. During data extraction and critical appraisal, 22 studies were excluded for not meeting 1 or more inclusion criterion; therefore, 27 studies were included in the final set for analysis. Figure 1 shows the PRISMA flow diagram for these results. For a summary of key findings, refer to online only Table 1.

Figure 1.

PRISMA flow diagram

Study characteristics

Publication information

Of the twenty-seven studies selected for inclusion, twenty-six were journal articles and one was an academic conference proceeding [14]. Sixteen articles, representing more than half of the publications in this set, were published in library and information science journals: seven in the Journal of the Medical Library Association [15–21], four in Medical Reference Services Quarterly [22–25], four in Health Information & Libraries Journal [26–29], and one in the Journal of the Canadian Health Libraries Association/Journal de l'Association des bibliothèques de la santé du Canada [30]. Ten articles were published in health sciences journals, including two in BMC Medical Education [31, 32], three in the Journal of General Internal Medicine [33–35], and five in other health sciences journals [36–40].

Country of origin

The majority of studies were conducted in the United States, although studies from Canada [26, 28, 30], the United Kingdom [27], and Italy [29] were included.

Study participants

Upper-year medical students and residents accounted for the majority of participants in these studies. However, undergraduate programs in nursing [40], dentistry [26], occupational therapy and physical therapy [28, 39], respiratory therapy and physician assistant [39], and graduate-degree students in physician assistant [24] and health sciences research [27] were also represented. The number of participants and their assignment to groups (e.g., intervention and control groups) varied considerably, from 10 [16] to more than 300 [25], with an estimated average of 50 participants per training session.

Length and year of study

Fourteen of the 27 studies were conducted over more than 1 year [17, 18, 20, 23–25, 30, 32, 33, 35–37, 39, 40], and 13 studies completed data collection within one year [14–16, 19, 21, 22, 26–29, 31, 34, 38]. Six studies did not indicate the year or years when the study was conducted [14, 19, 21, 27, 31, 38]. Four of the studies were conducted in the 1990s [15, 16, 33, 36], 12 occurred from 2000–2009 [17, 18, 22, 23, 26, 29, 32, 34, 35, 37, 39, 40], and 2 from 2010 to the present [28, 30]. The remaining 3 studies occurred over several years between 2006 and 2010 [20, 24, 25].

Evidence-based practice (EBP) skills addressed in instruction

The 27 included studies addressed 1 or more steps in recommended processes for EBP [1, 2]. Acquiring the best evidence (search strategy development, searching techniques, information sources, study selection, and acquiring of full text) was the most common step included, with all 27 studies addressing this skill during instruction. Asking an answerable question, clinical question development, and/or the PICO question format was second, with 22 of the 27 studies including this skill [15–20, 22–37]. Twelve studies [17, 18, 20, 22, 25, 29, 31, 32, 34, 36, 38, 39] addressed critical appraisal of evidence for 1 or more of the following: level of evidence based on hierarchy of evidence; study design; statistics use and reporting; therapy, diagnosis, or other question type–specific indicators such as likelihood ratios, number needed to treat, or absolute risk reductions; or evaluation criteria for qualitative and quantitative study designs, or for websites. Application of evidence to clinical cases, clinical scenarios, or case studies was addressed by 11 studies [17, 18, 22, 30, 32–36, 38, 39]. Assessing how efficiently and effectively the other steps were done was addressed in instruction by only 1 study [18].

With regard to instruction of the EBP process as a whole, one study [18] included the full EBP process. Five studies [17, 22, 32, 34, 36] included instruction in four out of five skills (all except assessment), and another seven studies [20, 25, 31, 33, 35, 38, 39] addressed three EBP skills, though which three of the five varied. For details on EBP steps covered in each study, refer to online only Table 1.

EBP information sources and tools included in instruction

EBP instruction in these studies overwhelmingly focused on searching the health sciences journal literature databases, with all but two [18, 26] of the twenty-seven studies teaching MEDLINE. The Ovid MEDLINE interface was utilized more often than PubMed, with a few studies utilizing both interfaces for instructional purposes. Several did not specify which interface was used. As many of the studies taught multiple tools, Figure 2 outlines the frequency of all online resources that were taught as part of EBP instruction as well as the corresponding study numbers. The Cochrane Library followed MEDLINE as the second most frequently taught resource in 10 studies. The remaining resources represented a variety of clinical medicine tools, EBM publications, and other health sciences journal literature databases, such as CINAHL and PsycINFO. The latter were taught less often, which is consistent with the target discipline for the instruction (medicine) and for the majority of the studies selected for this review. EMBASE was not taught in any of these studies on EBP instruction.

Figure 2.

Frequency distribution of resources taught in evidence-based practice

* Studies referencing the sources shown: DARE [32]; DynaMed [20]; InfoPOEMS [25]; AccessMedicine [20, 25]; PsycINFO [20, 24]; ACP Journal Club [32, 35, 39]; CINAHL [24, 39, 40]; MDConsult [22, 25, 39]; Clinical Guidelines [20, 22, 29, 38]; UpToDate [20, 32, 35, 39]; Cochrane Library [19, 20, 24, 25, 29, 32, 34, 35, 38, 39]; MEDLINE [14–17, 19–25, 27–40].

Methods of instruction

The majority of studies used two or more teaching methods, with only seven studies relying solely on one method [16, 17, 23, 24, 29, 32, 38]. However, some studies were not clear if only one teaching method was employed because teaching methods were not consistently described. Figure 3 depicts the instructional methods used in the studies and corresponding study numbers. The EBP instruction predominantly included lectures, also referred to in the studies as formal presentations or seminars, with twenty studies using this method. Small group discussions and/or one-on-one instruction followed, with sixteen studies using this method. Hands-on practice for online database searching in a computer lab was used in fifteen studies. Online instruction was much less commonly employed, with only six studies using this method, two of which used online teaching as their only method of instruction.

Figure 3.

Methods of instruction*

* Studies using each method: lecture [14, 15, 17–23, 25–28, 31, 33, 34, 36, 37, 39, 40]; computer lab [15, 19–22, 25, 26, 28, 29, 32, 35–37, 39, 40]; small group or 1-on-1 [14, 16, 18, 20, 21 (peer assessment), 22, 25–27, 30, 31, 33–36, 39]; online [17, 23, 24, 29, 32, 38].

† Combinations not used: lecture and online; lecture, computer lab, and online; lecture, computer lab, small group/1–on-1, and online; computer lab and online.

Image icon credits: “Multiple users silhouette” (small group) and “Computer class room” were designed by Freepik <http://www.flaticon.com/authors/freepik> via Flaticon, used under attribution license; “Lecture” and “Monitor” (online) were designed by ClkerFreeVectorImages, CC0 public domain release, via Pixabay.

While this systematic review focused on determining the best teaching methods in librarian-led instruction on EBP, only one study actually compared more than one teaching method. Schilling's presentation in 2011 at an e-learning conference compared traditional in-person EBM instruction to online EBM instruction. Students who received the online intervention had marginally better search scores than those who received traditional instruction. This difference approached but did not achieve statistical significance [14].

Assessment of EBP instruction

The overwhelming majority of the studies utilized one or more individual-level skills assessments, primarily of search performance, with only one utilizing a brief, classroom-level assessment [23]. Pretests and posttests of search skills were frequently reported [14–18, 20, 22, 25–27, 31, 32, 37, 39]. A variety of measures exist, several of which are standardized, such as the Berlin Questionnaire [41], the Fresno test [42], and the Evidence-Based Practice Questionnaire (EBPQ) [43]; however, very few studies [18, 19, 29, 31] utilized these standardized measures, making comparisons of study results across the studies unfeasible. Assessment tools designed in-house by the researchers in individual studies or taken from earlier studies and modified were more commonly employed. Self-reporting on attitude, confidence, or skill level was also commonly utilized as an assessment method [14, 16–20, 23, 26, 27, 32–35, 38, 39], primarily pre- and post-surveys of participants' attitudes and confidence in their EBM skills and course evaluations. Only those studies that employed these self-reports in tandem with other skills data collection methods were included in the review.

Role of librarians in EBP

Overall, librarians were involved in four aspects of EBP instruction with an almost equal distribution between planning curricula [14–18, 20, 22–28, 30, 31, 33–40], delivering instruction (all 27 studies), assessing student performance [14–30, 33, 35–40], and authoring journal articles on their EBP training experiences [14–28, 30, 31, 33–40].

Study design and statistical methods

Randomization of study participants was implemented in several studies [14, 16, 19, 21, 26, 29, 32, 33, 35, 38], with six studies clearly self-identifying as randomized controlled trials [16, 19, 29, 32, 33, 35]. The method of randomization differed between studies. Regardless of whether randomization was utilized, many studies had a clearly defined intervention group [14, 16, 19, 21, 26, 29, 32, 33, 35, 37–40], which received EBP instruction in comparison to the control group, which was not given the EBP instruction. Blinding was utilized in ten studies [14, 16, 19, 26, 29, 31, 33, 35, 38, 39], although what was being blinded and for what purpose varied across studies. The cohort design with pretests and posttests was frequently employed [14–18, 20, 22, 25–27, 31, 32, 37, 39, 40] with the intention of finding a significant difference in EBP skills, for example, search performance after the EBP instruction. Both search performance and attitudes toward EBP skills training were measured.

The overwhelming majority of studies included descriptive statistics, only two studies did not [15, 28]. Seven studies reported only descriptive statistics [16, 20, 22–24, 28, 40], and one study noted the use of inferential statistics but did not report data or statistical results [36]. Reported statistics consisted of primarily percentages and to a lesser extent, measures of central tendency (mean, median) and deviance (standard deviation, interquartile range). Many of these studies presented one or more inferential statistical tests to determine the significance of any differences found in posttest versus pretest scores or intervention group scores versus control group scores. Popular parametric measures of group differences reported were the t-test for differences between two groups [14, 19, 26, 27, 32, 34, 37–39] and analysis of variance (ANOVA and its variants) for differences between more than two groups [26, 27, 31, 32]. Non-parametric tests for group differences were also reported in several studies: McNemar change test [15, 18], “non-parametric Wilcoxon” and Spearman's p [39], Wilcoxon signed rank test [18, 25, 33], Wilcoxon rank sum test [21, 33, 35], Fischer's exact test [33], chi-square test [14, 21, 34, 36, 38], and Mann-Whitney U test [17, 29, 32].

The level of significance varied from 0.05 to 0.0001. Several studies achieved high significance (p<0.001 or less) on at least 1 measure [17, 18, 25–27, 29, 31–35, 38, 39]. Good significance (0.001<p<0.02) was achieved on at least 1 measure [17, 18, 26, 27, 29, 32, 33, 35, 38, 39], and minimum significance (0.02<p≤0.05) was achieved for at least 1 measure in several studies [15, 17, 19, 26, 27, 31, 32, 34, 35, 37, 38]. Two studies showed a trend toward significance but did not achieve significance (0.06<p<0.1) on at least 1 measure [14, 21]. Effect sizes for inferential statistics were not reported for many studies; thus, the research team does not know the magnitude of the differences between groups for each study. Overall, statistical tests reveal weak positive differences between intervention group versus control group or posttest versus pretest.

DISCUSSION

Based on the results of this systematic review, it was not possible to determine which EBP teaching method was most effective. No previously published systematic reviews focused specifically on the effectiveness of instructional methods for teaching EBP with librarian involvement in health sciences curricula. There were systematic reviews that analyzed general information literacy skills training in the health sciences and the most effective methods for teaching information literacy skills to undergraduate students [12, 44, 45], but these studies did not address EBP instruction or directly focus on the role of the librarian in teaching EBP. Other literature reviews summarized EBM instruction in undergraduate medical education but excluded other health professional students and did not discuss the role of the librarian in teaching EBM [46, 47].

Publication information

It is worth noting that sixteen of the twenty-seven articles in this review were published in library and information science journals. The academic conference proceedings article was written by a library school faculty member. While the designations of the authors did not always clearly identify librarian involvement, only three articles were written solely by librarians (who were not also clinicians) with no clinician or researcher coauthors [14, 20, 24]. The remaining articles, regardless of the journal, involved librarians, clinicians, and/or researchers, or did not clearly indicate if a librarian was involved as a coauthor.

Methods of instruction

Studies have shown that teaching EBP skills to health professional students results in an increase in knowledge and skills [48–50]. Library instructors often utilized a combination of teaching formats, usually lecture combined with lab practice and/or small group discussion (18 out of 27 studies), suggesting that the instructors felt that the use of a combination of teaching methods was the best approach. In fact, there were indications that the most effective form of EBP instruction was when content was delivered over multiple sessions utilizing several teaching methods [40] and that one-shot single sessions might not be the best option to teach information literacy skills [51].

Koufogiannakis and Wiebe sought to determine the most effective methods for teaching information literacy skills to students at the undergraduate college level. Using fifty-five studies that either compared two teaching methods or instruction to no instruction, they found that though each of the teaching methods was more effective than no instruction, no particular method was more effective than another [12].

Only one study in this systematic review compared more than one teaching method. Schilling's presentation in 2011 at an e-learning conference compared traditional in-person EBM instruction to online EBM instruction. Study participants were first-year medical students (n=128). The control group (n=63) received traditional, instructor-led training in skills in information and MEDLINE searching, while the intervention group (n=65) participated in the same instruction via a web-based tutorial. The e-learning module was structured to exactly duplicate the classroom instruction. Students who received the latter intervention had marginally better search scores than those who received traditional instruction. This difference approached but did not achieve statistical significance [14].

The findings suggest that search performance improved regardless of instructional method, but due to the large variability in study sample, sample size, measurement tools, and statistical tests employed, the research team was unable to quantify the improvement. In a systematic review where the study included all patrons of academic libraries, nine out of ten studies did not show any difference in learning between computer-assisted instruction and traditional lecture-type instruction [52]. With no consensus in the literature about which instructional methods are most effective for teaching EBP, future studies are needed to investigate and compare teaching methods in this area.

Assessment of EBP instruction

In this systematic review, all twenty-seven studies used a performance-based skills assessment or evaluation of student learning, as this was one of the inclusion criteria. Only four studies [18, 19, 29, 31] used a standardized, validated assessment tool for skills performance assessment: the Berlin Questionnaire [41], used by one study [18], and the Fresno test [42], used by three studies [19, 29, 31]. The EBPQ [43] was used by one study [19] for a self-reported assessment of skills, attitude, or comfort. The most popular method of assessment was found to be the pre/post format of giving tests before and after an intervention. Instructor-developed rubrics, modifications of standardized assessment tools, and intervention-control skills testing were also used. In a 2006 systematic review, Shanyefelt et al. summarized and classified assessment instruments used to evaluate EBP skills [53]. A key criteria for inclusion in their study was validity of the test, which was lacking in the majority of papers in this study. It is unclear whether the lack of use of validated instruments was because there was no assessment tool that was sufficiently robust to meet the needs of health librarians' EBP teaching methods or program curricula or if awareness of the availability and applicability of these instruments was not sufficient.

Role of librarians in EBP

In the studies included in this review, librarians held an instructor role in all twenty-seven studies. Librarians were also involved in other aspects of EBP instruction from developing curricula, planning and teaching EBP material, to assessing student progress and contributing to the scholarly literature.

The nature and extent of the librarian role seems to be influenced by several factors. For example, authorship in this set of included articles seems to correlate with the role of the librarian in the study in question. In general, those authors who published in library science journals usually describe in detail the role of the librarian in all aspects of instruction. The majority of these authors are academic health sciences librarians. In contrast, similar studies published in medical journals are primarily authored by medical faculty, with one or two librarian coauthors, and the librarian role is primarily that of EBP instructor; although librarians are also involved to some extent in curriculum planning and student performance assessment in these studies as well.

Depending on the circumstances and opportunities, librarians can do well in any role and contribute at all levels in the EBP teaching process [49]. Teaching information skills seems to be the most common role for librarians in EBP instruction, and research has indicated that librarians may have better learning outcomes for one-shot, face-to-face information literacy sessions than nonlibrary instructors [51]. As information skills are very important for EBP, librarians should be encouraged to participate, at the very least, in the information skills step of any EBP instruction. Faculty involved with health-related curricula should be strongly encouraged by administrators to invite librarians to be part of the EBP teaching team.

Finding opportunities to include library EBP instruction in already busy school curricula, medical curricula in particular, may present a challenge [51]. Depending on circumstances, web-based online courses could be considered as an alternative, because a well-designed online course may be successfully taught and result in better learning outcomes compared to one-shot lecture-type instructions [14, 51].

Study design and statistical methods

For this set of twenty-seven studies, findings were generally weakly positive, showing an improvement in search performance. Several studies demonstrated robust positive findings for better search performance or positive attitudes toward EBP skills training or both, while others obtained mixed findings. Large variability in study sample, sample size, measurement tools, and statistical tests employed made quantifying the amount of positive change in search performance and other measures of EBP skills by meta-analysis impossible. Almost all studies examined whether instruction on EBP provided by librarians (and clinicians or clinical faculty) led to a measurable change in participants' skills and/or attitudes toward EBP. Only one study addressed the topic of this systematic review—which instructional method is most effective for librarians teaching EBP—and the findings showed a trend toward significance for the intervention group, which received online instruction, versus the control group, which received traditional instruction [14].

Limitations

The decisions regarding the search process may have introduced some bias in the studies found and selected for this review. Although a few non-English language databases were searched (Spanish, Portuguese, Japanese), many others were not included, potentially creating language bias. Hand searching of journals related to the research topic was not conducted, and therefore, some relevant studies may have been overlooked. Meeting posters and presentations were searched, but they were ultimately excluded for lack of complete data on quantitative learning assessment, which was one of the inclusion criteria. Also, in cases where information was unclear or not included in the actual article, authors were not contacted for additional information, which could have affected the data analysis. For example, details on the grading of assignments not included in the actual article, but retrieved from authors, could have clarified or improved the analysis of learning assessment.

Biography

Stephanie M. Swanberg, MSI, AHIP, swanberg@oakland.edu, Assistant Professor and Information Literacy and eLearning Librarian, Medical Library, Oakland University William Beaumont School of Medicine, 100 Library Drive, Rochester, MI 48309; Carolyn Ching Dennison, MA, MLIS, AHIP, cdenniso@hawaii.edu, Science and Technology Librarian, University of Hawai‘i at Mānoa Library, University of Hawai‘i at Mānoa, 2550 McCarthy Mall, Honolulu, HI 96822; Alison Farrell, MLIS, alisonr@mun.ca, Public Services Librarian, Health Sciences Library, Memorial University of Newfoundland Health Sciences Centre, 300 Prince Philip Drive, St. John's, NL, Canada A1B 3V6; Viola Machel, MLIS, BMSc, vmachel@stcatharines.library.on.ca, Librarian, St. Catharines Public Library, 54 Church Street, St. Catharines, ON, Canada L2R 7K2; Christine Marton, PhD, christine.marton@utoronto.ca, Adjunct Instructor, Faculty of Information, University of Toronto, 140 St. George Street, Toronto, ON, Canada M5S 3G6; Kelly K. O'Brien, MLIS, kkobrien90@gmail.com, Assistant Professor and Regional Assistant Librarian, Crawford Library of the Health Sciences–Rockford, University of Illinois at Chicago, 902 North Main Street, Apartment 11, Rockford, IL 61103; Virginia Pannabecker,§ MA, MA (IRLS), AHIP, vpannabe@vt.edu, Assistant Librarian (start of study), Health Sciences Librarian, ASU Libraries, Arizona State University, Phoenix, AZ; Mindy Thuna, MSc, MISt, mindy.thuna@utoronto.ca, Research Enterprise Liaison Librarian, University of Mississauga Library, Hazel McCallion Academic Learning Centre, University of Toronto Mississauga, 3359 Mississauga Road, Mississauga, ON, Canada L5L 1C6; Assako Nitta Holyoke, MD, PhD, MSLIS, holyokea@slu.edu, Professor and Medical Reference Librarian, Medical Center Library, Saint Louis University, Learning Resources Center, 2nd Floor, 1402 South Grand Boulevard, Saint Louis, MO 63104