Exploring the Complexities of Experimental Design: Using an On-line Reaction Time Program as a Teaching Tool for Diverse Student Populations

Abstract

Students are rarely given an opportunity to think deeply about experimental design or asked to develop experimental protocols on their own. Without participating in these endeavors, they are often unaware of the many decisions necessary to construct a precise methodology. This article describes an on-line reaction time program, and how I have used this program as a teaching tool for students to explore experimental design. This approach can be tailored to meet the level of any undergraduate student – from non-science majors to upper-level biology/psychology/neuroscience majors, affording all students the opportunity to think like a scientist. Described is how I use the reaction time program for a whole class demonstration and discussion, as well as, how it can be used for a written assignment in which each student designs and conducts his/her own experiment outside of the classroom. Comments from several students, who did the written assignment, are included to provide a sense of their thoughts and considerations. When students are given a simple method, such as the measurement of reaction time, it allows them to focus exclusively on developing precise methodology, which taps into types of thinking that they are not often asked to exhibit in other science classes.

INTRODUCTION

Inquiry is fundamental to how experimental science is conducted, yet, exposing undergraduate students to this process can be challenging, especially when teaching courses that do not have an associated laboratory section. Many students are not familiar with how to develop a testable hypothesis or they may believe that they do not know enough about scientific methods to design an experiment. Indeed, student misconceptions and inaccuracies regarding randomization, sample size, and proper controls have been described at the college-level (Anderson-Cook and Dorai-Raj, 2001; Hiebert, 2007), graduate-level (Zolman, 1999), as well as in professionals publishing in the life sciences (Festing, 2003). However, by using a simple experimental measure, students can become engaged in the process of scientific inquiry, and in turn, begin to think deeply about experimental design. As an example of the power of this approach, this article describes how I have used an on-line reaction time program with diverse groups of students as a means to have them explore issues surrounding experimental design.

The measurement of reaction time has been used for many years in physiological psychology. Factors such as attention, fatigue, or the use of central nervous system stimulants or depressants affect an individual’s reaction time, whereas reaction time between individuals varies depending on age or motor skill practice (Kosinski, 2009). Measurement of reaction time has recently come to the forefront of our cultural consciousness. For example, a year ago The New York Times ran a series of articles entitled “Driven to Distraction” highlighting the dangers imposed by drivers who engage in activities that take their focus away from the road (Richtel, 2009). In turn, neuroscience research is beginning to uncover evidence that our brains are not as adept at multi-tasking as many people may think or believe themselves to be (Clapp et al., 2010). Taken together, the measurement of reaction time is quite topical for today’s science classroom, especially since advances in information and communication technology have radically altered our ability to stay focused on a single task (Richtel, 2010). In this article, I describe how I have used an on-line reaction time program as a teaching tool for students to explore experimental design. I also share quotations from several students who reflected on the process of using the program to conduct their own experiments outside of the classroom.

On-line Reaction Time Program

The on-line reaction time program that I use consists of a virtual red-yellow-green traffic light (http://getyourwebsitehere.com/jswb/rttest01.html). At the start of each trial the yellow light is illuminated. To begin, the subject must click an on-screen button to the right of the traffic light. Each time this button is clicked, the red light is illuminated for a variable amount of time, up to seven seconds. When the red light turns off and the green light is illuminated the subject must click the on-screen button as fast as possible; doing so registers the subject’s reaction time. The program runs five trials, and displays both the raw data (values for each trial), and computes the average reaction time to a thousandth of a second. The program is easy to use, and is readily available to anyone with internet access, which makes it ideal for use both inside and outside of the classroom. On the web page, just below the reaction time test, the person who developed the program (Jim Allen) includes a statement of permission for general use, along with several tips and caveats for its use.

Inside the Classroom: Demonstration

I use the on-line reaction time program as part of a whole class demonstration and discussion. I have done this demonstration six times over the past two years with different groups of students: summer research students from local community colleges or elite four-year colleges/universities, and undergraduate students at my own institution enrolled in Introduction to Biology (for non-majors) or Neurobiology (upper-level biology majors). Before starting the demonstration, we discuss the concept of reaction time in a way that corresponds to the scientific level of the audience. For example, when students do not have much/any background in neuroscience, reaction time is described simply as sensory input leading to motor output, and is differentiated from a reflex. For more scientifically sophisticated students, I have the class outline the neural pathways involved - from visual stimulus to voluntary motor response. The key is that I adjust the explanation to the scientific level of the audience so that students are given just enough background to be able to use the program as an experimental measure.

Next, I ask the class to choose one variable to test and together we construct a specific hypothesis. Then, we outline how to conduct the experiment. Within this basic framework, there is a great deal of improvisation since different factors associated with experimental design come to the forefront with each group of students. However, certain issues are always touched upon, including: the nature of the experimental design (within subject or between subject) and who the ‘ideal’ subjects would be for the experiment. The depth and scope of the demonstration are also affected by the amount of time I have to work with. For example, when I have had a generous amount of time (90–120 mins), I divided the students into small working groups, and asked each group to devise their own hypothesis and experiment. These working groups then reported their ideas to the class, and the class chose a single variable for the demonstration. However, when I have had less time (50–60 mins), there was no small group work beforehand; I solicited ideas from the entire class before we decided on a single variable to test. In this way, the demonstration is adaptable to meet the needs of the students, and the amount of time available to the instructor.

In my experience, each group of students responds differently to the challenge of choosing a variable for the demonstration. Students in my Neurobiology class were quite vocal, offering many reasonable suggestions, so the issue was simply deciding on a single variable to test. In contrast, students with less science background (summer students from local community colleges and non-majors in Introduction to Biology) seemed more reluctant to respond to such an open-ended request. When a class was particularly quiet, I found it helpful to ask questions about their interests and habits – focusing on qualities that might have an effect on reaction time. For example: How many students are left handed or right handed? How many students play video games? If so, which types of video games – action or strategy? When do students use their cell phones? Are they ever talking on their cell phones while they engage in another activity requiring their attention? By asking specific questions, student responses can serve as a spring-board to selecting a variable to test. However, regardless of the variable chosen and the process by which it was chosen, I have noticed that even the quietest groups of students become vocal and engaged with the demonstration once they were asked to outline how they would conduct their experiment.

As the class develops its methodology, I guide the students by asking questions. The specific questions are based on the chosen variable and hypothesis, which differs for each class. This process flows into having several test subjects (student volunteers) run through the experiment in front of the class. This is an important part of the demonstration. It is at this point that the students’ abstract ideas about how to do the experiment are challenged; they are now forced to articulate all of the steps involved in their methodology. This process alone is an eye-opening experience for them, as they are rarely asked to work out the details of a method in their other science courses. For example, with the very first subject, students notice immediately the role of practice (learning) in improving the consistency of an individual’s reaction time (Kosinski, 2009). Consequently, as they work out the details of their methodology they must decide how to control for this ‘learning effect’ – such as having each subject practice the test several times prior to conducting the actual experiment (Anderson-Cook and Dorai-Raji, 2001).

After running several test subjects, together we look at both the raw data and the average reaction time values determined by the program. This leads to a discussion of variability (individual and group), and the magnitude change necessary for a variable to have an effect on reaction time. The students are usually surprised by their preliminary data, especially when they differ from their initial predictions. For example, students had expected to see large differences in reaction time between the dominant and non-dominant hands of individuals, but our preliminary data did not support this conclusion. However, when another class tested whether experience with action-type video games affected reaction time, our preliminary data suggested that avid video game players were consistently faster than subjects who did not play video games. Another variable with a large effect was verbal distraction (answering a series of random questions aloud); this condition drastically increased the reaction time of individuals compared to when the same subjects did the test without verbal distraction.

While I do not do statistical analysis as part of the demonstration, it would be easy to do so, either during class or as a homework assignment. The beauty of using a simple (and quantitative) method, like measuring reaction time, is that it can be tailored for the purpose of the instructor. While my interest is in having students explore experimental design, if another instructor were interested in having the students think deeply about data analysis and interpretation, the same demonstration could be adapted to focus on that aspect in more depth.

Overall, I feel that the keys to implementing this demonstration successfully in the classroom are (1) clear goals, and (2) time management. The instructor should decide, at the outset, which aspect he/she wants to emphasize: hypothesis, methodology or data acquisition/analysis. My focus has been on methodology. As a consequence, I typically have only enough time to run 2–3 subjects during the demonstration, which means we cannot examine our data in any depth. However, if an instructor wanted to have a large enough sample size for statistical analysis, he/she would want to devote less time to having students develop their hypothesis and methodology during the demonstration. This could be accomplished several ways: by the instructor actively guiding the experimental design process during class, making it more streamlined, or by assigning this material as homework prior to the demonstration. Alternatively, the instructor could use one class period for the students to develop their hypothesis and methodology, and the next class period to collect data from a large enough number of subjects to permit detailed data analysis (Hiebert, 2007).

Outside the Classroom: Written Assignment

I have taken the reaction time demonstration a step further in my Neurobiology class. This is an upper-level elective for Biology majors, with an enrollment of 70–80 students. Approximately half of the students take the course as ‘lecture-only’ for reduced credit; this option is available since the laboratory sections cannot accommodate everyone who wants to take the course. Over the ten years I have taught Neurobiology, I have tried various types of assignments for the ‘lecture-only’ students in an effort to give them inquiry-based learning experiences. For the past two years, I have used the reaction time demonstration as a spring-board to have these students use the on-line program outside of class to design and conduct their own experiment with friends/family as subjects. Their written assignment is a 300–500 word scientific abstract, which includes introduction, method, results and conclusions within a single paragraph. Overall, the student work has been very good; they were able to choose a single variable to test, and were able to devise a precise methodology using the on-line reaction time program. Most notably, I found that even students who performed poorly on the course examinations could do well on this assignment.

While it was clear from reading the students’ abstracts that they were able to do this assignment successfully, I wanted to gain better insight into their thought processes while carrying out their work. Therefore, last year I required the students to submit brief written responses, along with their abstract, which described: (1) examples of specific issues that they had to consider in order to conduct their experiment, and (2) what they learned about the process of science by doing the assignment. Interestingly, many students reported struggling with issues that I had hoped they would have to confront regarding the precision and consistency of their methodology. I have selected several apt quotations from student responses that I feel speak directly to this particular issue.

Student Comments about the Assignment

One student, who compared reaction time in subjects of different ages, reported, “…I learned that coming up with consistent methods is more difficult than I thought. In my labs I’m used to going through the methods that have been worked out previously without appreciating the science behind the development of those methods.” This student goes on to say, “I also learned that there is a lot of work between hypothesis and conclusion. Hypothesis is the easy part (though even that has to be developed correctly so as to be testable). Then there is the development of the methods and getting the right people in the right places for testing and carefully interpreting data.” A second student, who examined the effect of caffeine consumption on subjects’ reaction time, echoed these sentiments by noting that “Much more effort/time may be needed to set up the experiment rather than perform the experiment…..An experiment that is not well-controlled or organized will yield data but the data itself may not be useful.” A third student, who examined whether listening to music (compared to silence) affected subjects’ reaction time, reported that the “…biggest challenge was consistency. Although the online test was, in fact, online and while that makes for convenient subject recruitment, it also introduces another variable: environment.” This student goes on to say, “To fix this, everyone used my personal Macbook in my bedroom, with no one else in the room but myself. The data could be misconstrued if it were different computers/mice and different environments. This ensured that it was a neutral setting with no preference.” A fourth student, who also examined the effect of music on reaction time, reported “I learned that the process of science requires attention to unexpected details in order to reduce extraneous variables; for example, I thought I would play the same music for everybody, but actually since tastes differ, people would have reactions of pleasure or displeasure or even emotional reactions, so it was better to simply ask the subjects to choose the music they normally listen to and which does not incite extreme emotional reactions.”

What I feel comes across in these comments is the profound attention to detail that each student gave to designing and conducting his/her experiment. There is a distinct sense of ‘ownership.’ By asking students to do the work of scientists, they now have a greater sense of what it means to be a scientist. And by using a simple experimental method, they can focus their thoughts almost exclusively on how to conduct their experiments in a controlled, reproducible manner. I would like to end with a quotation from a fifth student, who studied the effect of hand dominance on subjects’ reaction time; in commenting on what was learned about the process of science, this student reported “That what you think will happen going into any experiment isn’t necessarily how your results are going to turn out, no matter how sound your reasoning or how strongly you thought it would turn out to be true.” The power of this assignment is that it helps students move from the realm of the abstract to testing empirically what they thought might be true. By doing this they are forced to confront their assumptions, and examine their data with care in order to determine whether their predictions were supported or not.

CONCLUSIONS

The use of a simple method, such as the measurement of reaction time, allows students the opportunity to think deeply about experimental design. Too often in our courses the methodological details are worked out beforehand. While this aids in the execution of experiments in a timely manner, it removes an important learning component as students struggle intellectually to develop their own methodology (Anderson-Cook and Dorai-Raj, 2001; Hiebert, 2007). The approach outlined in this article brings experimental design to the forefront. While I have focused on the measurement of reaction time, the same approach could be adapted for any simple method, such as determination of the blind spot in each eye or two-point touch discrimination. The key is to use a method that is as simple as possible, so that students can turn their attention to how, precisely, to perform the experiment. This exercise provides a meaningful and lasting learning experience for science students of all levels.