Use of instructional design theory and an individualized hybrid strategy for assessment in library-based instruction

BACKGROUND

Since 1996, a librarian at the University of Florida, Gainesville, has partnered with a professor to develop and implement a term project in PCB3063, an undergraduate genetics course with first-day enrollment of approximately 120 students. Evaluative studies suggest that the class and project represent successful collaborations between the library and an academic unit [1, 2], and students report a high level of satisfaction with the course on standard course evaluations. One unique aspect of the course project is that topics are individually assigned to students, and assessment, including assessment of database searching skills, is tied to that specific topic. While course instructors are convinced (and the literature supports [3]) that such individualized assignments enhance student learning, grading the assignments for this large service course required approximately 45 hours of effort per semester for the collaborating librarian. In 2008, the instructors (professor and librarian) explored ways to decrease grading time without compromising student learning and with the intention of retaining the individualized assessment.

ASSIGNMENT OBJECTIVES AND ORIGINAL FORMAT

The PCB3063 term project requires each student to use the primary literature and databases of factual information to explore the genetic underpinnings of a specific assigned disorder. Since the inception of the project, students have completed three incremental searching assignments (parts A, B, and C) that provide a road map for locating information and incorporate the concepts of database searching and critical appraisal of the literature [1].

To support these assignments, students receive 4 hours of hands-on genetics and literature database instruction from the librarian (Table 1), as well as an introduction to the use of journal house style and critical appraisal of the literature. At the end of the first 2-hour session, each student is assigned a unique genetic disorder and, throughout the semester, individually completes searching assignments parts A–C. These assignments are worth nearly 37% of the final project grade and prior to 2009 were graded manually by the librarian.

Table 1.

Characteristics of assessments part A, B, and C

As the librarian's job responsibilities became more administrative, it became necessary to explore options for reducing the time she spends on PCB3063. Because the hands-on instruction had proved to be beneficial to students, no reduction in the amount or format of instruction was considered. However, due to the customized nature of parts A–C and the 45 hours required to manually grade them, students did not receive feedback in a timely manner. Such a delay was a considerable problem, as students should not attempt later assignments until they receive feedback on the earlier ones, and research supports the increased effectiveness of prompt feedback [4]. It was therefore determined that any changes related to the searching assignments would occur at assessment, rather than during instruction, with the expectation that the instructors would utilize a course management system for assessment.

MOVE TO PAPER-ONLINE HYBRID ASSESSMENT

Moving paper-based assignments to the online environment provided challenges and required reflection on expected learning outcomes (Table 1) [5–7]. The paper-based assignments assessed learning through fill-in-the-blank and short-answer questions. Such questions had to be transformed into multiple-choice and true/false options for automatic grading to occur, without compromising any student learning that might occur during assessment and without affecting assessment quality. Formative computer-based assessment can facilitate concept retention [8] as well as subject-related mastery [9]. Well-designed multiple-choice questions can extend beyond mere testing of fact-based knowledge and instead require students to interpret, synthesize, or apply understanding. However, the individualized nature of 120 assignments—requiring Online Mendelian Inheritance in Man (OMIM), Medical Subject Headings (MeSH), and genetic attribute identification—gave the instructors pause: How could such assignments be graded online? Additionally, how would search strategies or critical appraisal be assessed in the online environment? Given the diversity of tasks students were expected to perform, it was clear that a one-size-fits-all approach for assessment would not work. It was decided that a hybrid approach, with assignments consisting of both print and online components, would satisfy student learning as well as streamline grading. Students would continue to work through the original paper-based parts A–C, with some search components and critical appraisal graded manually. The majority of student work, however, would be assessed through WebCT.

The focus of part A is the identification of the correct OMIM record and MeSH term, and the assignment is easily transferred to an online environment. For example, the paper version of the assignment asks students to supply the: “Official title (name) of condition exactly as listed in OMIM.” In the online environment, the question is offered in multiple-choice format. A student assigned Multiple endocrine neoplasia, type 2A (MEN 2A), would find these possible answers:

A. Thyroid carcinoma, familial medullary, included

B. Thyroid carcinoma, papillary

C. PTC tasting, included

D. Multiple endocrine neoplasia, type IIA

E. Multiple endocrine neoplasia, type IIB

While creating 120 online quizzes for part A was time consuming the first semester (eighteen hours), in subsequent semesters, quizzes need to be updated only if the MeSH term or OMIM information has changed. Transforming part A for electronic grading was the most onerous aspect of the move to the online environment; however, the ability for students to receive immediate feedback and move quickly on to part B was an important enhancement. Another complication is encountered when creating such simplified multiple-choice questions covering database searching. Students have the ability to work backward, essentially searching each listed answer until the correct one is identified (learning nothing), rather than completing the assigned search to determine the correct answer. To minimize this behavior, online assessments are timed (five minutes for part A), and students are encouraged to work out their answers on the paper version (short answer and fill in the blank) before starting the timed online assessment.

Part B concentrates on literature and genetics database searching and critical appraisal of the literature, and provided different challenges during transfer. In addition to answering fill-in-the-blank and short-answer questions, students were required to submit their search strategies, which were graded manually. Such grading was a brutally slow process (approximately twenty hours), especially given the numerous and “creative” search strategies students employed. In the move to online grading, it was imperative that students' understanding of search concepts still be assessed. Therefore, students continue to complete the paper version of part B and perform searches. The online assessment is translated into a multiple-choice format as in part A, but with an emphasis on general searching concepts, rather than individual search results.

For example, the paper version of part B asks students to search for the genetic aspects of their assigned disorder using MeSH and then print their search strategy. When completing their fifteen-minute assessment on WebCT, students find the following questions:

The correct format for your search is:

A. Your MeSH term AND genetics

B. Your MeSH term [MESH] AND genetics [MESH]

C. Your MeSH term [MAJR] AND genetics [MAJR]

D. Your MeSH term/genetics [MESH]

E. Your MeSH term and genetics [MESH]

Which of the following is true concerning MeSH searching?

A. MeSH acts as a dictionary and helps the searcher find the official search term to use for a particular concept

B. Using a MeSH term retrieves all papers on a particular topic, regardless of how current the paper is

C. Once you have found a MeSH term, you may then throw out peripheral papers and focus on a particular aspect of a topic (such as genetics or therapy)

D. A and B

E. A and C

Because these are universal concepts, all students can complete the same part B online assessment. Students also turn in the paper-based part B, as those questions that address critical appraisal of the literature and that require information from students' specific GenBank records can only be graded by hand.

In part C, search strategies take a backseat to biological information. This assignment is a challenge from a logistical standpoint, as the molecular details of some assigned disorders are not well understood [10]. Nor does every assigned disorder appear in GeneReviews, a resource covered in part C. Therefore, students first explore these databases with known biological entities (for example, hemoglobin) and then attempt them with their assigned genes and proteins. Using the hybrid approach, students complete their searches and answer questions on the paper part C form. The twenty-minute online component asks questions related to the known biological entities, rather than those related specifically to their assigned disorder, again allowing all students to complete the same online assessment. For example, students might be asked to perform the following search:

A zinc finger is a structural motif within a DNA-binding protein that allows binding to occur. Search NCBI's Structure database for 3-dimensional structures of mouse proteins that contain zinc fingers. Narrow this search such that you find only the structures that were submitted by Pabo and colleagues in 2001. What is the PDB number for the structure that you find?

Such a question can easily be transformed into a multiple-choice question for the online environment. Following the online assessment, students submit print outs of their annotated three-dimensional protein structures for manual grading.

OUTCOMES

Part A is now graded entirely online, saving approximately 10 hours of grading time, providing immediate feedback to support learning [4], and allowing students to quickly move on to part B. Seventy percent of part B and 80% of part C can now be graded online, saving approximately 15 and 12 hours of grading, respectively.

Although the time savings are essential for continued librarian collaboration in the course and immediate feedback obviously benefits students, the instructors are adamant that course changes must not in any way hamper student learning or assessment quality. Because the major instructional intervention (4 hours of face-to-face instruction) and the homework assignments themselves (paper parts A–C) have not changed, there is no reason to hypothesize that student learning is compromised by the introduction of online grading for a portion of the searching assignments. Conversely, an argument could be made that the online-graded, multiple-choice questions in part B that require conversion of student search strategies into more conceptual statements might facilitate additional learning by requiring students to translate, interpret, synthesize, or apply understanding. However, a comparison of grades for parts A, B, and C from the 3 semesters before and 3 semesters after the move from paper-only grading to hybrid assessment shows no significant differences in student performance. Across all 6 semesters, student scores for parts A–C do not differ significantly, according to their broadly overlapping 95% confidence intervals. More specifically, mean student scores between the 3 semesters before versus after the move differ by less than a point (4.88 versus 4.73, 21.10 versus 20.48, and 23.25 versus 22.87, respectively). Although it is impossible to state unequivocally that student performance on assessment (paper or online) equates to true learning, the evidence suggests that student learning has not been influenced for better or worse by the addition of online assessment. Additionally, the fact that student scoring patterns have not changed suggests that creating well-considered multiple-choice questions and using the combination paper and online assessment systems can retain assessment quality.

Because individualized assignments are conducive to student learning [3], it was essential that the project retain them in the hybrid assessment environment. However, doing so increased the paper to hybrid migration workload. Initial migration for parts A–C took approximately twenty-two hours, with the majority of this time spent developing the multiple-choice questions and answers, rather than the physical act of inputting them into WebCT. Even so, the initial migration resulted in a net gain of fifteen hours, as using automated grading saved approximately thirty-seven hours. Subsequent semesters require only three to four hours to accommodate changes in MeSH terms or OMIM information and to incorporate new disorders, genes, or proteins. In 2011, the university adopted Sakai as its course management system. The information technologists in the Department of Biology performed the physical migration, leaving only a few hours of clean-up work for the librarian.

CONCLUSIONS

Moving classroom assessment to the online environment requires more than a simple transfer of questions from the print. More importantly, it also requires consideration of the core expected learning outcomes. Questions can then be carefully translated using instructional design principles to retain those learning outcomes. The instructors of PCB3063 relied on such principles during their translation of parts A, B, and C, resulting in a hybrid online and paper approach. This approach permitted the continued use of individualized assessments, important in student learning. Translated assessments stressed student comprehension regarding the search process and the transfer and application of knowledge from the specific to the general contexts [11].

By moving to the online environment, students receive more timely feedback and the burden of grading has been greatly diminished. Since no changes were made to the instructional intervention (face-to-face instructional sessions) or the assignment content, no change in student performance was expected, and none was detected. The similarity in assessment scores before and after the development of the hybrid assessment system suggests that the quality of the assessment instrument has not changed, even though the mode of delivery has. Student performance and feedback suggest that the move to the hybrid system was successful in achieving the instructors' goals: retaining student learning, while saving the librarian and professor considerable time. For these reasons, the use of instructional design theory in developing online assessment is recommended to others considering similar course enhancements.