There Is More than Multiple Choice: Crowd-Sourced Assessment Tips for Online, Hybrid, and Face-to-Face Environments

ABSTRACT

Developing effective assessments of student learning is a challenging task for faculty and even more difficult for those in emerging disciplines that lack readily available resources and standards. With the power of technology-enhanced education and accessible digital learning platforms, instructors are also looking for assessments that work in an online format. This article will be useful for all teachers, but especially for entry-level instructors, in addition to more mature instructors who are looking to become more well versed in assessment, who seek a succinct summary of assessment types to springboard the integration of new forms of assessment of student learning into their courses. In this paper, ten assessment types, all appropriate for face-to-face, blended, and online modalities, are discussed. The assessments are mapped to a set of bioinformatics core competencies with examples of how they have been used to assess student learning. Although bioinformatics is used as the focus of the assessment types, the question types are relevant to many disciplines.

INTRODUCTION

Life science educators have responded to the recommendations in Vision and Change (1–3) to improve undergraduate education by developing new courses and programs in bioinformatics, biomechanics, systems biology, and other emerging, interdisciplinary fields. These initiatives reflect that quantitative measurements, advanced technology, and data science are becoming increasingly important in biology. However, many instructors are struggling to implement these changes due to a lack of preparation time, resources, and assessments aligned with updated instructional competencies and learning resources (4). Further, recent attention to issues of justice, equity, inclusion, and diversity requires faculty to employ creative ways to fairly assess all learners (5).

Supported by new technologies, higher education has experienced a rapid evolution in available teaching modalities (e.g., blended, HyFlex, synchronous, asynchronous online) (6). The widespread adoption of online teaching, most recently spurred by the COVID-19 pandemic, will remain an important part of undergraduate education; therefore, student assessments will need to be innovatively structured for online, face-to-face, and blended environments (7). There is thus an immediate need for assessments that are valid, reliable, and flexible in new learning environments.

Challenges associated with curriculum development and adapting to new teaching modalities present an opportunity for instructors to implement the core Vision and Change action items by aligning assessments to learning goals and integrating multiple forms of assessment to track student learning (1–3). To help instructors implement new assessment tools or refresh current assessment strategies, we have prepared this summary of assessment types that are appropriate for multiple learning environments. Although bioinformatics, an emerging interdisciplinary field, is the theme, the assessment types are widely applicable to other fields. The provided example assessments are mapped to bioinformatics core competencies (8) to model how assessments can align to learning outcomes that include both concepts and skills.

PROCEDURE

The Network for the Integration of Bioinformatics in Life Sciences Education (NIBLSE) is an NSF-funded Research Coordination Network for Undergraduate Biology Education (9, 10). NIBLSE has established a set of bioinformatics core competencies for undergraduate biologists (Fig. 1) and is working to provide vetted bioinformatics learning resources (4, 10, 11). The NIBLSE Assessment Validation Committee (AVC) compiles, reviews, and aligns assessments to these core competencies. Although not an exhaustive list, the summary presented here describes 10 assessment types used regularly in undergraduate teaching by NIBLSE members and other bioinformatics faculty (Appendix 1 in the supplemental material). All question types have been used in face-to-face, blended, and online modalities and were submitted by NIBLSE steering committee members and instructors who completed a survey (4). Here, we provide a brief summary of each type and discuss trade-offs, along with providing a crowd-sourced exemplar of a bioinformatics-based assessment aligned to a student learning outcome and a core competency for an undergraduate course (Appendix 1 in the supplemental material).

FIG 1.

The nine NIBLSE bioinformatics core competencies for undergraduate biologists. See reference 8 for the full description of each competency.

CONCLUSION

Within the context of effective assessment, it is important to consider two features: validity and reliability (12). Considering these two features here is timely, as a recent analysis of the quality of bioinformatics assessments found that <1% of studies assessing student learning gains mentioned the use of both validity and reliability measures (13).

Validity relates to actually measuring what one seeks to measure. For example, if bioinformatics is the stated focus of a test, the test would not be valid if it only addressed basic biology concepts. There are various ways to measure validity and different types of validity, such as “content validity” (an assessment measures the targeted content of a field of knowledge adequately and sufficiently), “construct validity” (an assessment measures the intended knowledge or skills), and “concurrent validity” (an assessment that correlates well with a previously validated instrument) (14). A simple initial step to help ensure content validity is to have colleagues in the same field review and critique an assessment. Construct validity can be tested with a small group of novice students verbally describing their interpretation of assessment questions.

Reliability relates to how consistently a test produces the same scores when taken by similarly prepared students. There are various ways to demonstrate reliability, such as “test-retest,” “internal consistency,” and “parallel forms” (15). Typically, reliability is demonstrated by giving a particular test two or more times, while looking at how consistent the results of a test are when students have not had additional learning interventions. Useful statistical procedures for examining reliability are provided by the Web Center for Social Research Methods (16).

Importantly, assessment questions should strive to discriminate between higher and lower levels of cognitive learning according to Bloom’s taxonomy (17). It is also important to separate out those questions that contribute effectively to the overall assessment and those that lower overall assessment reliability. A common strategy that is often built into Learning Management System (LMS) environments is the item discrimination index (Fig. 2) (18). This is a correlation coefficient (point-biserial based) which ranges from −1 to 1. The magnitude and sign of the index for a given question reflect how well that question discriminates between high- and low-scoring students; a positive value indicates that high-scoring students tended to answer the question correctly, while low-scoring students didn’t, and vice-versa. A minimum acceptable correlation coefficient threshold of 0.15 is suggested, with good items generally performing at >0.25 (19). Questions performing lower than the minimal threshold should be reviewed or refined for wording, presentation, and context.

FIG 2.

Screenshot of bioinformatics assessment results of two different multiple-choice questions. The question in panel A has a higher discrimination index than that in panel B, which means that it is more effective at discriminating between high- and low-scoring students. The question in panel B has a relatively low index, which suggests that the question is actually intruding upon the objective and indicates that high-performing students may be confused or “tricked” by that question. Lower discrimination indices (<0.25) are often labeled in red by the LMS to alert that the question may need to be reviewed or refined.

It was obvious from the examples submitted that instructors are striving for strong and innovative assessments aligned to a set of core competencies. However, it was also clear that creating effective assessments takes considerable time and effort, as NIBLSE instructors who contributed assessments often qualified their examples as “drafts” or “evolving.” Here, we provide an overview of assessment types and encourage the reader to further explore the rich literature on assessment in the STEM classroom, starting with Handelsman et al. (20) and Dirks et al. (21). These 10 crowd-sourced assessment types and accompanying summary provide instructors with a quick reference for designing aligned assessment instruments independent of classroom instructional modality.