An online team-based practical exam in lieu of final reports for a design course

Challenge

Anecdote suggests that the abrupt transition of university courses to wholly online instruction was especially difficult for engineering design courses. Design courses have evolved toward team-based activities, hands-on building and testing, and the availability of fabrication equipment and other facilities. As the pandemic set in, instructors sought meaningful online opportunities to interact with students in their BME design courses, for students to work together as teams, and means for providing fair and meaningful assessments.

We here address summative assessment in a team-based design course and suggest that an online practical exam can in many cases be a preferable alternative to final design papers or comprehensive final exams, regardless of whether the class is delivered online or in-person. Indeed, there is evidence that among the several options for assessment – selected response, extended written response, performance assessments, and direct communication [1] – practical exams (a form of performance assessment) are better measures of student learning than are written reports [2] – a common staple of engineering design assessment. The most important and challenging factor in assessment is to match the assessment method to the intended goal [1].Performance assessment seems to be a clear choice for a class whose goal is to instill the hands-on skills needed to engage in the engineering design process for clinical devices. We are aware of no reports on the use of practical exams during remote or online learning, and few reports of practical exams in biomedical engineering courses [3].

Novel initiative

This initiative was realized in a second-year design course in BME that focused on the development of skills needed for later steps in the engineering design process – namely detailed design, prototyping, testing, and some elements of communication. This course has been described elsewhere in greater detail [REFERENCE REDACTED FOR BLIND REVIEW]. Briefly, Individual training was provided in a variety of fabrication techniques and other skills that we judge to be valuable to the engineering design process as it pertains to medical devices. These included:

1. Embedded controllers and basic electronics (“Arduino Uno Ultimate Starter Kit + LCD Module”, Vilros.com)

2. CAD, mechanical drawings, and FE simulation (Autodesk Fusion 360)

3. 3D printing

4. Soldering and de-soldering of circuits

5. Forming and shaping soft materials (band saws, jig saws, drill presses, sanders)

6. Reference management software (Zotero)

7. Efficient use of word processing features (Microsoft Word)

Students were required to purchase the embedded controller learning kit in lieu of a textbook for this course. All of the above learning was completed and assessed on an individual basis, not a team basis.

The class culminated in a 2-week closed-ended team-based design problem – to design and build a simple intravenous infusion pump (peristaltic pump) using the provided components of a microcontroller project board, stepper motor, motor driver, and LCD display with push buttons. Students designed and fabricated all the components that were not provided to them and wrote the necessary firmware to control the pump and display relevant infusion parameters. There were nominally six students per team.

This course has a typical semester enrollment of 65 students, 50% of whom are female, and is taught twice per year.

In 2017 the final assessment in this design course was transitioned from a final design report to a team-based practical format that was administered during the final exam period. The goal was to acquire a more meaningful assessment of their skills, and to engage students in a consequential learning experience during the exam period. Student teams worked to complete three challenges: using microcontrollers to a new purpose, performing CAD with assembly and FEA simulation, and conducting digital image analysis. They were permitted to divide those tasks as they saw fit among their team members, and they had to complete their work as a team within the three-hour final exam period. It served as a realistic measure of some of the skills they had acquired, and their ability to work as a team. The instructor and teaching assistants circulated through the room to answer questions. The teams submitted their work online by the end of the exam period.

The team-based practical was retained in the transition to online instruction in Spring of 2020. Zoom was used with pre-assigned breakout rooms for teams to work during the final exam period. Students were able to call the instructor or TAs to their breakout rooms when they had questions. One significant variation was needed from the in-person practicum format, that being that students were allowed twelve hours rather than three to complete their work. This change was made to accommodate team members who were working from outside the local time zone. Despite this option, most teams elected to complete their work within the customary three-hour time block, possibly because the instructor was available to answer questions and circulated among the breakout rooms during the customary three-hour block, but not outside that time.

The three challenges on the practical were identical to that offered in Fall of 2019. They were:

1. Re-create the parts of a commercially available cane in CAD, combine them in an assembly, and perform a finite-element analysis to determine whether it will support the load claimed by the manufacturer.

2. Using a microcontroller and parts, create a circuit to detect the brightness of a laboratory environment that contains light sensitive compounds. Display the light level on an LCD, and display a warning and sound an alternating tone when a light level is exceeded.

3. Using ImageJ, determine the number, mean size, and mean contents of fluorescent particles in an image containing thousands of them – some of them not visible to the naked eye.

These challenges closely resembled the types of problems that students had been solving during the semester. The microcontroller portion of the practical was graded “live,” since it was to design a circuit and code to a specification. It either worked as specified or it did not, earning either full credit or no credit. The CAD and microcontroller challenges were graded after the fact with partial credit scoring.

There was no significant difference in team scores on the final practical exam comparing Spring of 2020 to Fall of 2019 (p=0.60 by two-tailed t-test. N=11 and 13, respectively). End of course evaluations were completed before the practical and so cannot be used as a point of comparison for purposes of this paper.

Reflection

From the perspective of the instructor the online practical exam was easier to manage than in years when they were conducted face-to-face. The classroom environment during the face-to-face practical exam is chaotic and loud, and it was sometimes difficult for teams to get the attention of the instructor, or for the instructor to distribute their time equitably. The online environment, while disconcertingly quiet, provided students a much more equitable share of the instructor’s time.

Troubleshooting of circuits proved surprisingly easy to do by asking students to hold their breadboards up to their webcams. The ability of students to share control of their computer with the instructor proved useful for guiding them in the CAD component of the practicum. In fact it has been the experience of the author, and others [4], that online instruction in CAD is just as effective as learning by other means. It is therefore not surprising that assessment would also be equivalent between mediums.

This approach offers sufficient advantages over the in-person team-based practical exam to consider retaining the online practical exam even after the pandemic comes to a close. Practical exams in general offer the additional advantage of reducing test anxiety since they focus attention on familiar principles and practices [5]. This might complement the reduction in test anxiety that some have reported in online exams compared to paper exams [6], [7].

Successful implementation of a practical exam in design is subject to caveats. First, it is critical that practitioners offer extended time relative to in-person instruction for these experiences and allow for asynchronous work. This not only satisfies the needs of students in different time zones, but also helps to mitigate the effects of communications failures (which were many) and provides a small but necessary measure of equity for students who lack broadband. Second, pre-assigned breakout rooms are critical in a class of any substantial size in order to cut down on the time needed to connect students with their teams. Unfortunately, many students have personal accounts in addition to university-provided accounts for teleconferencing. Even with warnings to make sure that students are signed on through the correct account, practitioners have to allow time to do manual sorting and re-assign students to breakout rooms whose internet connections failed.

The team-based practical exam is not without its potential shortcomings. Of greatest concern is the tendency of students on the team to specialize. Those who are best at CAD work on the CAD portion of the exam, and likewise for the other sections. As Schmidt writes, “Student specialists can work together to produce a great project without adequately learning all project elements. Educators have failed to notice that strategies students use to achieve good project outcomes may be undermining individual learning outcomes.” [8] While the team-based practical exam may lead to better results overall and better represent how things are done in the workplace, it may not adequately assess individual learning or promote individual learning.