Developing a Peer Review Process for Web-based Curricula

Abstract

The World Wide Web creates new challenges and opportunities for medical educators. Prominent among these are the lack of consistent standards by which to evaluate web-based educational tools. We present the instrument that was used to review web-based innovations in medical education submissions to the 2003 Society of General Internal Medicine (SGIM) national meeting, and discuss the process used by the SGIM web-based clinical curriculum interest group to develop the instrument. The 5 highest-ranked submissions are summarized with commentary from the reviewers.

The World Wide Web (the web) contains approximately 3 × 1012 pages—roughly 30 pages for every person who has ever lived.¹ The web is attractive to medical educators because it allows rapid updates of information; inclusion of multimedia; distribution of information over large geographic areas at little expense; and—for better and worse—dissemination of ideas unencumbered by traditional mechanisms of publication and peer review.

Ambiguity about the evaluation of web-based educational curricula has stifled discourse in the field. Currently available criteria for critique of health information sites focus on sites directed at the general public. These criteria, although analogous to ours as regards content, generally lack the specific attention to evaluation, feedback, and learner-oriented material that is especially important in medical education.^2,3 The absence of an accepted tool for qualitative assessment of web sites specific to medical education makes it difficult to share and be recognized for high-quality work.

However, this has not arrested investment in web sites. The Clerkship Directors in Internal Medicine list 27 institutions with sites dedicated to the internal medicine clerkship⁴—and that list is incomplete. Twenty-four percent (33/135) of the Innovations in Medical Education (IME) abstracts at the American Association of Medical Colleges (AAMC) 2003 meeting were related to web-based curriculum.⁵ Because web sites are not “published” and no royalties are returned, many medical schools use password-protected private sites, which limit collaboration and resource sharing. If evaluation and dissemination of high-quality web sites are improved, resources can be directed more efficiently.

The web creates new challenges for evaluation. An excellent teaching web site should provide information—the “content” of the site—of quality comparable to traditional print materials (such as journal articles, textbooks, and syllabi). However, the web offers more options for presentation than print media. These include the ability to “link” from one page to another by clicking on relevant text, the inclusion of animations and sounds, and the capacity for interactive pages that modify content based on user responses. These innovations in presentation put an additional burden on evaluators, who must consider both content and format when critiquing web sites.

The Society of General Internal Medicine's (SGIM) web-based clinical curriculum interest group, founded in 1999 by 3 of the authors (HAS, BLH, DED), was established to foster collaboration among internal medicine educators in the development, maintenance, and evaluation of web-based curricular materials. The perceived need for improved recognition of web-based materials among the group members spurred the creation of a subcategory in The Innovations in Medical Education poster session for web-based materials at the 2003 SGIM national meeting. We sought to 1) develop a process and peer evaluation tool for critically reviewing and comparing content and aspects of presentation (links, multimedia, interactivity) that are unique to the web, and 2) provide an opportunity for SGIM to help developers receive academic credit for their efforts.

METHODS

The group reviewed the literature on distance learning,⁶ Internet curricula,^7,8 and differences in communication between print and web, as well as student evaluations of our own educational sites.⁹ An instrument for evaluating health information on the Internet¹⁰ was identified and reviewed. In addition, one of us (DED) was involved in developing the SGIM peer review process for UpToDate,¹¹ and we drew on that model.

The interest group discussed peer review at SGIM in 2001 and 2002, and the authors met via teleconference in 2002 to draft a preliminary document. Seventeen evaluation criteria, organized into six categories, were constructed as the initial instrument. Input was solicited from two domain experts on peer review and three on web page design to provide feedback on the evaluation criteria, in an effort to establish content validity. The instrument was then pilot tested using existing educational web sites created by three of the authors (HAS, BLH, DED). Modifications were made to simplify the instrument and address issues of clarity and usability. The instrument was submitted to the SGIM IME committee and was further revised, based on their feedback, to more closely parallel the review criteria used for SGIM abstract review, while maintaining criteria unique to the web. Fourteen evaluation criteria organized into 3 categories (content, format, and evaluation/ feedback) were eventually agreed upon. A Likert scale was used for the scoring system for each of the 10 criteria. This process culminated in the instrument that was used to review web-based submissions to the 2003 SGIM national meeting.

A total of 15 abstracts and web sites were reviewed in a nonblinded manner by 6 reviewers with the instrument described below. Reviewers were volunteers from the interest group; 3 reviewers had previously reviewed for SGIM in other categories. Reviewers did not evaluate sites that they had submitted, or in which there were other conflicts of interest. While the web sites were rated using our scoring criteria, they had to be adapted to fit the SGIM program committee's COS 3-item instrument that asked reviewers to rate submissions based on importance of question, rigor of methodology, and conclusion, with no elucidation of the criteria for these categories. We used our rating system as the formal criteria for COS scoring. Sites with the highest scores (cutoff set at the 50th percentile) in the COS reviewer system were chosen for presentation; 11 (74%) were presented, compared to 80% of the 1,292 total 2003 SGIM submissions. Although reliability was not formally assessed, our COS rankings were similar to other IME submission categories.

Final judging was done onsite at the IME presentation sessions during the 2003 SGIM national meeting. Two judges from the review committee, neither of whom had sites being presented, used the instrument below to score presentations. The scores were averaged between the two judges to produce a final ranking. Following the instrument, we present the 5 highest scoring sites in that final onsite ranking.

THE INSTRUMENT

The evaluation instrument is seen in Table 1. The 5 highest scoring submissions to the 2003 SGIM annual meeting are summarized in Table 2.

Table 1.

The Evaluation Instrument

Criterion	Score
Content	Strongly Agree	Agree	Uncertain	Disagree	Strongly Disagree
The material presented is up to date and accurate	1	2	3	4	5
The curriculum material is appropriate for the specified learners	1	2	3	4	5
The references are appropriately cited	1	2	3	4	5

Format/Presentation	Strongly Agree	Agree	Uncertain	Disagree	Strongly Disagree
The format is appropriate for the learning objectives	1	2	3	4	5
Links are necessary and add to understanding the subject matter presented	1	2	3	4	5
The site is interactive and engages the learner	1	2	3	4	5
A multimedia format is used effectively (e.g., audio or video clips, pictorial clinical material)	1	2	3	4	5
Multimedia elements are appropriate (not excessive) and enhance learning of the subject matter	1	2	3	4	5
The pages, links, and multimedia elements load at reasonable speeds	1	2	3	4	5
The site is easy to navigate and appropriately layered to allow browsing and quick access to material	1	2	3	4	5

Evaluation/Feedback	Strongly Agree	Agree	Uncertain	Disagree	Strongly Disagree
There is a method for the learner to evaluate the site	1	2	3	4	5
E-mails or links available for feedback to the authors/web master	1	2	3	4	5
There is a method to evaluate the learner	1	2	3	4	5
There is a method for the learner to receive feedback on their comprehension of the material presented	1	2	3	4	5

Specific themes: content

Is the material presented up to date and accurate?

The ease of updating web-based material makes the requirement for current, accurate content more stringent than that expected for print media. Dates of last update should be available.

Is material cited appropriately?

This includes citations for literature and multimedia content. Ideally, each reference should link to the original source or its MEDLINE abstract.

Specific themes: format/presentation

Do links contribute to the educational value of the site?

The number and content of links, a defining element of the web, can enhance or diminish a site. Examples of appropriate links are those to relevant peer-reviewed literature or practice guidelines; inappropriate links are irrelevant, poorly validated, or commercial/promotional material.

Is the site interactive?

Examples of interactivity include: a simple multiple choice quiz with feedback on correct and incorrect answers; a case scenario in which one receives clinical information based on one's diagnostic choices; or a physical examination simulator, where the mouse is used to “listen” to various points on the chest wall. Simple media (e.g., movies) are not necessarily interactive.

Is the site well organized?

Web sites are best when organized for the medium.¹³ Student feedback indicates that simply “dumping” print material onto the web is unsatisfactory. Users read material in short chunks and appreciate transparent organization.

Does multimedia contribute to the site's educational value?

Animations, video, and audio are among the most exciting strengths of web curricula. However, longer page load times and the need for more expensive, powerful computers limit the use of multimedia. Well-used multimedia enhances the content of the site, including auscultatory sounds, an echocardiogram, and animation showing how to read an X-ray. Poor uses of multimedia such as frivolous animations or sound effects distract the user and slow site performance.

Specific themes: evaluation/feedback

Is there a method for the learner to evaluate the site?

The ability to change online materials quickly is best exploited when there is continuous feedback from users to facilitate improvement. Learners (and reviewers) should be able to contact the author, editor, or web master with questions or comments about the site.

Is there a method for the site to evaluate the learner? If so, does it provide feedback to the learner on their performance?

Interactive sites can incorporate evaluations of learner comprehension. Although this can be time consuming, learner evaluation can provide valuable information to the site authors about the success of their educational effort, and offer feedback to the learner on their knowledge and performance.

Table 2.

The Five Highest Scoring Submissions to the 2003 SGIM Meeting

URL
Educational Purpose	Audience	Content	Format	Evaluation/Feedback	Limitations
http://www.pharmapac.org Osterberg LG, Stiller C-O, Ayers M, Tornqvist E, Gustafsson LL. Stanford University and Karolinska Institutet, Sweden.
To provide self-directed learning in clinical pharmacology	Advanced medical students	Case-based approach to pharmacotherapy. Provides guidance in use of the web to find answers.	Good use of multimedia in cases. Interactive question format.	Quizzes after each case; pretest before the course and a posttest given after the course.	Requires site registration; pre- and posttest data not available.
http://medicine.osu.edu/exam Krieger CG, Knutson D, Cain T, Fish C, Finneran C, Gabel LL. Ohio State University.
To teach the fundamental principles of physical examination	Medical students and other health professional students	Simulated physical examination of eight primary content areas	Digital video clips, interactive multimedia elements, and instructive text. For example, the learner can click on an image of a chest and hear breath sounds.	Immediate feedback given regarding the performance of certain aspects of the physical examination.	Online site is demonstration for CD-ROM product. Requires Flash Player to view.
http://www.docsynergy.com Kendrick D. Tulane University.
To deliver an online interactive educational experience	Community physicians	Lectures recorded from multiple sources and converted to online media	Delivers lecture-based teaching materials to remote sites using readily available technology.	Students can take quizzes online and scoring and awarding of credit occurs immediately.	Most content not authored by site developers. Limited interactivity.
http://courses.washington.edu/med665 Dewitt D. University of Washington.
To aid in teaching the core medicine clerkship at multiple rural sites	Third-year medical students	Over 84 problem-based cases are presented with questions linked to explanatory materials.	Multimedia cases with interactive questions. Links to sites such as clinical guidelines are provided.	Immediate feedback is given, tailored to the student's answer. Users evaluate the site via a built-in evaluation tool.	Minimal use of multimedia. Case format varies widely between cases.
http://endeavor.med.nyu.edu/courses/pps/courseware Kalet AL, Gany F, Changrani J, Steinberg S, Lee A, Nachbar M. New York University School of Medicine.
To teach fundamental skills for interviewing patients with limited English proficiency	First-year medical students	Focused on knowledge, skills, and attitudes needed to effectively interview patients with limited English proficiency	Students view a video montage describing the use of medical interpreters. Students then critique 6 video interviews of limited English proficiency patients.	Pretest and posttest format. Students compare their responses to expert analysis. Students evaluate the program after completion.	Password protected for institutional access only.

DISCUSSION

The web offers medical educators an exciting new opportunity to expand curricula. Its advantages include interactivity, accessibility, and timely updates. However, it has historically been outside the scope of traditional mechanisms of peer review. Previous studies in this area have focused primarily on evaluation of content rather than use of teaching methods effective in medical education. One recent article,¹² which did look for adherence to specific didactic methods (critical thinking, evidence-based learning, independent learning, and feedback), was limited by strict inclusion criteria (only 24 of 112 sites met criteria for review) and use of a dichotomous scale that was not designed to differentiate between high-quality sites. Of the 24 sites reviewed, they found that only 17% used all 4 teaching methods. We present a new model for critical evaluation of web-based medical curricula with the hope that it will identify and differentiate excellent teaching web sites, help web authors achieve recognition for their work, and ultimately result in general improvement in quality of web-based education.

Our work has several limitations. We report the first attempt to develop a rating instrument for web-based medical education curricula. We used standard grounded theory approaches to developing the instrument, including literature review and expert consultation, and believe the resulting instrument has good face validity. While our rankings paralleled those of other SGIM abstract submissions, other formal tests of instrument reliability, such as internal consistency (Cronbach's α), factor analysis to assess construct validity, or interrater reliability measures, were not done. Such testing should be pursued as an important next step in providing a reliable peer review process. The SGIM web site peer review cluster should consider assessing several aspects of this instrument's reliability in the future, as the sample size grows. SGIM's web group might consider developing a committee to do formal web site educational based review, as a means of providing members scholarly credit. It would also be interesting to test this instrument in other noninternal medicine medical educational curricular web sites, to demonstrate generalizability.

Another significant difficulty in our undertaking was the substantial number of sites with restricted access, usually to members of the sponsoring institution. The question of whether peer-reviewed web sites should be publicly disseminated, as are other peer-reviewed materials, remains controversial and should be debated further. If web sites are to remain restricted, a mechanism should be worked out whereby their contributions to medical education can be promulgated without making the entire site open to the public.

Our evaluation criteria offer a standard peer review method for web-based curricula. We hope that this will facilitate communication, discussion, and collaboration in the field, and provide a foundation for future research.