A course for teaching and learning about the responsible conduct of research

Abstract

Over the past 30 years, the scientific community has been increasingly challenged to provide the next generation of researchers with training in responsible conduct of research (RCR). Although RCR courses, workshops, and seminars are now routinely taught internationally, there is little uniformity in goals, content, pedagogy, duration, class size, or methods of assessment. The result is a mixed picture of effectiveness. In this manuscript, we describe goals, rationales, and features for a course tested and revised through well over 100 iterations. Based on our experience and that of others with whom we have shared this model, we propose this course as one that RCR instructors might readily and successfully adopt or adapt.

Introduction

In the US, teaching of responsible conduct of research (RCR) has been increasingly expected since first required for trainees supported by National Institutes of Health (NIH) training grants (National Institutes of Health, 1989). Because it was from the NIH, this early requirement was of particular relevance to biomedical research, but over the past 40 years RCR requirements have been increasingly recognized as relevant to most if not all forms of scholarship. Many graduate programs now require such training for all of their students, a few institutions require RCR training of all researchers, and in response to the America Competes Act of 2007 the National Science Foundation (National Science Foundation, 2009) requires that undergraduate students, graduate students, and postdoctoral scholars funded by National Science Foundation (NSF) awards receive RCR training. In parallel with efforts in the US, research communities elsewhere in the Americas, as well as in Asia, Europe, and Africa have also increased their focus on the need for training in responsible research, often modeled after or similar to the US (e.g., Abdi et al., 2021, Barr et al., 2020, Rossouw et al., 2014).

Unfortunately, one of the risks of unfunded mandates is that financially constrained research institutions will opt for a nominal compliance framework, rather than building on a perception that RCR education is fundamental to good science. One way in which compliance framing is exhibited is an unwavering focus on the topics recommended by NIH (e.g., National Institutes of Health, 2009; Office of Research Integrity, 2000). A risk of an unquestioning adherence to this list forecloses real engagement with the myriad and complicated ethical questions raised by the practices of science/research. Beginning in the early days of implementation of these requirements, there have been calls for a re-imagining of the approach, and for more meaningful and expansive RCR education (Dubois & Dueker, 2009; Mastroianni & Kahn, 1998; Pimple, 2002; Steneck & Bulger, 2007), but more recent examinations of how this training is actually being implemented show that institutions are still missing the mark when it comes to effective and robust RCR education (Phillips et al., 2018; Resnik & Dinse, 2012). For example, in a study of effectiveness of RCR education (Antes et al., 2009), based on published findings (i.e., a sample strongly biased to be the strongest RCR programs), the authors concluded that “ethics instruction is at best moderately effective as it is currently conducted.” In a more recent study, Phillips et al. (2018) found that 82% of universities surveyed relied on online-only training for RCR instruction. Similarly, based on a recent study in the European Union, aspirations of required training were not being met with an effective understanding and awareness of the principles defining responsible research and innovation (Novitzky et al., 2020).

It is also clear that context and circumstance matter. For instance, in the face of the COVID-19 pandemic and renewed calls for social justice, we are challenged to reckon with blind spots and with uncritical devotion to a static idea of what the research enterprise looks like and how it functions. This opportunity is precluded if we restrict teaching to an unchanging list of topics or an unvarying model. Instead, we would benefit from the flexibility to respond, for example, to current events with a focus on manifestations of social inequity in our research environments and results, the lack of public trust in science, and the importance of transparency, openness and inclusion in all aspects of the research process.

What follows is a description of the on-going development of an RCR course which has taken these realizations into account. The course as currently designed and facilitated is offered as a model that might be useful for others not only in the US but internationally in designing RCR courses. More specifically, the hope is that we are generating an RCR course that satisfies a requirement but at the same time is a meaningful, student-centered learning experience.

Method

In response to the 1989 NIH requirement (National Institutes of Health, 1989), an RCR lecture course was developed at UC San Diego. The audience from the beginning has been and remains primarily graduate students in the first couple of years of their program, with some postdocs and occasional faculty. Because of NIH requirements, these individuals tend to come from diverse biomedical disciplines, but with increasing requirements from NSF and some graduate programs, these courses frequently include people from biological sciences, engineering, and occasionally social sciences. The initial version of the course met 1.5 hours for each of ten meetings (total of 15 hours) during the Spring quarter starting in the early 1990s. The course name (“Scientific Ethics”) is one of many titles given to courses like this one, so for all practical purposes we use terms like “RCR,” “research ethics,” and “scientific integrity” interchangeably. As was true of many such courses of the time, the course was based largely on lectures delivered by the course instructor and guest lecturers followed by discussion of cases.

In subsequent years, experimentation with different approaches and a body of literature on effective teaching and RCR education stimulated an evolution of the course in terms of goals, content, pedagogy, duration, class size, and methods of assessing outcomes.

Results and Discussion

Goals

Beyond the nominal goal of meeting requirements for RCR training, there are many other credible goals for RCR courses (Kalichman & Plemmons, 2007; Kalichman, 2012). Recognizing that it is not practical to address all of these goals, we limited our focus to three high level goals as proposed in Kalichman (2007), and reframed those as learning objectives:

• To know rules, issues, options, and resources for research ethics

• To understand the purpose and value of ethical decision-making

• To have a positive disposition toward continued learning about research ethics

Topics

In the early years of the course, the topics covered were largely defined by NIH recommendations for RCR training (e.g., Kalichman, 2012). The lists of topics varied, but up until 2000 primarily included conflicts of interest, data management, authorship (and publication), research misconduct, and research with human and animal subjects (Kalichman, 2013). In 2000, the Office of Research Integrity briefly proposed a new RCR training requirement in which they expanded the list to include mentoring, collaboration, and peer review (Office of Research Integrity, 2000). Although that requirement only lasted for two months (National Institutes of Health, 2001), those 9 “core instructional areas” nevertheless became an expected standard for such training. A similar list was subsequently formalized as NIH guidelines (although not requirements) in 2009 (National Institutes of Health, 2009), but with the addition of societal responsibilities:

“the scientist as a responsible member of society, contemporary ethical issues in biomedical research, and the environmental and societal impacts of scientific research.”

These re-formulated nine topics dictated the focus of the UC San Diego course for several years. However, based on hearing the diverse contexts and experiences of our students, we found that limiting “RCR” to the nine specific topics articulated in the 2009 NIH notice was too narrow. Instead, we sought an approach that recognized the larger context in which practices of scientific investigation sit, and we also wanted to be sure that the conception of “scientific investigation” not preclude other forms of research and scholarship. As we experimented with different approaches, we came to understand the benefit to our students of considering a wider range of topics, speaking to the broader integrity of the research enterprise. This means not only the processes in the production of knowledge, but also with the application of that knowledge, and how both of those impact – and differentially so – a diverse “public.” In short, our approach was adjusted to allow for all the elements that help maintain the integrity of the research environment, writ large.

Pedagogy

This course was initially taught largely with the traditional model of an expert (the professor) giving lectures, answering questions, and possibly leading discussion with the class. However, subsequent reading of the literature about effective pedagogy provided evidence that optimal approaches depend on learning that is student-centered, structured around exercises that promote student engagement (e.g., Bransford et al., 2000; Brown et al., 2014). In the RCR field, at least back to the early 1990s, this was assumed to be defined by case-based teaching (e.g., Korenman & Shipp, 1994; Macrina & Munro, 1993). In our experience, this view is still widely promulgated at meetings on RCR education. However, in teaching multiple iterations of the course we noted at least two flaws in this argument. The first is that merely having a discussion about a case is insufficient. It’s necessary to structure those discussions in ways that allow students to generate their own ideas and understandings, modeling what might happen if they were to encounter something similar in their practice as researchers. Second, the benefit of discussing cases is that students become engaged in part because of the novelty of the exercise. However, that novelty is quickly lost if the same approach is used session after session. To address this challenge, we experimented with a wide variety of approaches (e.g., case-based discussion, debate, role play, etc.) and discovered that any of these approaches can stimulate thoughtful discussion, but that student enjoyment and engagement appear to be optimal when the approaches vary (Kalichman & Plemmons, unpublished data, 2008). This is consistent with the concept of “desirable difficulty” according to scholarship on teaching and learning (Brown et al., 2014). In short, we tend to learn best not when we repeat the same task again and again, but when we have to struggle to address the task in a variety of different ways.

Class size

Through the early 2000s, the number of students required to take this course increased, resulting in sections of as many as 60 students by 2008, with commensurate reduction in the quality of discussion, student engagement, and accountability. Based on the perception that an instructor could better connect with students in smaller classes, an arbitrary decision was made that classes should be divided so as to have no more than 20 students per class. While many factors might determine the optimal class size, it seems self-evident that in a class based on student interaction, smaller class sizes are more likely to favor such interaction (e.g., McKeachie, 1980).

Meeting duration

Clearly, many factors might change optimal meeting times including, but not limited to, the learning objectives and methods of teaching, the size of the class, attitudes and abilities of the students, the skill and style of the instructor, and the culture of the institution. Our decision to make each class meeting two hours in duration was informed by three observations. First, student feedback in evaluations for a variety of similar courses convened for times ranging from 1–3 hours have resulted in mixed feedback, but there has been a tendency to consider classes too long at 3 hours and too short at 1 to 1–1/2 hours. Second, if the goal is to have up to twenty students engaging multiple times in discussions during the course of a class, and if it’s hoped each student would speak for at least a few minutes, then it’s mathematically clear that one hour would be too short. Finally, on the assumption that a single session could be divided into two blocks for different topics, it seems reasonable that one hour per topic would allow for nominal in depth discussion. Admittedly, all of these factors are based on subjective impressions, but the result is a course format that has been well received by the students, and yields productive conversations based on the perceptions of the instructors.

Group assignments

By 2010, it was clear that student evaluations for the course remained very positive, but that wasn’t necessarily evidence of effective, student-centered pedagogy. With a view to promoting more effective pedagogy, we adopted a variation on a practice common in research seminars and journal clubs in which participants take turns presenting and leading discussion. Starting in Spring 2011, using the NIH topics as a starting point, students were divided into groups and were either assigned or allowed to choose topics for presentations. Each student group was allotted one hour for their presentation. This allowed for presentations by ten groups over five meetings of two hours each. Each group was specifically charged with providing a brief introduction to be followed by various strategies and approaches characterized by engagement and interaction among the students in the course. In sum, the students choose specific approaches (e.g., debate, role play, think-pair-share) to maximize peer engagement and potential for effective learning. One example of the value of these forms of engagement was described recently as an approach to facilitate learning of effective strategies for ethical decision-making (McIntosh et al., 2021).

Group size

Our target for group size was informed largely by personal experience. On the one hand, by definition a “group” with just one member affords no opportunity for engagement and discussion in the process of preparing a presentation. At the other extreme, we observed cases in which groups of more than three or four members increased the chance that at least one member of the group could effectively leave the work to others. This isn’t to say that a much larger group, with the advantages of increased diversity of membership, couldn’t be a rich experience. However, to maximize the chance that all group members would participate, we chose to aim for two members per group, but occasionally allowing the group to include 3 or even 4 members if class size became larger than intended.

Group selection of topics

Although our courses initially tracked the NIH topics fairly closely, several problems with doing so were identified (Kalichman, 2013): (1) these topics are not really distinct as they often overlap, (2) the list of topics is not really specific guidance as one class on any given topic could cover entirely different material than another, (3) some things on the list were much less important to some researchers than others (e.g., human or animal subjects); and (4) some things as important or more important to some researchers were not on the list at all. For these reasons, we began to ask what other topics might be worth discussing, which resulted in a much longer list of possible topics (e.g., Kalichman, 2016). Given the realization that even the NIH recommended topics were not prescriptive as to what should be covered about each of those topics, it seemed better pedagogically to allow students to choose topics of particular interest to them from the larger list, or even to think of their own topics. The result is that the course isn’t designed simply to cover the topics recommended by NIH, though almost unavoidably the standard nine topics get covered in the introductory lecture and/or the weekly group presentations. Giving learners the responsibility for selecting their own topics might seem risky, but in our experience even new graduate students often have an interest in a topic or idea that would be a good fit. However, to facilitate topic choice, we simply provide a list of about 30 possible topics, encouraging students to choose one of those listed or to use the examples to come up with their own topic. By talking with the students, the instructor can easily adjust focus or approach. In our experience, adjustments are typically minor at most.

Faculty participation

The NIH guidelines for RCR training include an expectation that training grant faculty will be involved in the training. This arguably is both because faculty participation sends the message that this is important to established researchers and because their perspectives and experience can help to inform discussion. However, this goal cannot be met simply by having faculty show up as not all faculty will contribute usefully and because it isn’t realistic to have numerous faculty show up at each session in the hope of representing the many different areas of research defined by the students in a particular class. As a compromise, we experimented with and then adopted a model in which an open invitation is sent to faculty to join the final meeting of the course -- with a goal of no more than 3–4 faculty attending. Although not always successful, we seek diversity in the panels in terms of career stage, gender, discipline, etc. These panels have effectively become a capstone event consisting of wide-ranging discussions framed by the faculty answering questions raised by the students about RCR issues. It is noteworthy that these faculty are not joining the course to “teach” RCR. They are advised that their attendance requires no advance preparation other than their academic expertise and experience. The faculty in effect are simply talking about how they do research and how they handle (or would handle) the challenges raised by the students. These discussions are moderated by the course instructor, who invites the perspectives of multiple faculty, while engaging the students as part of the conversation. The result is almost invariably a learning experience for everyone, including the faculty, and rated very highly by the students as a valuable addition to the course. Taken together, just as student engagement helps to support a culture of ethics, incorporation of faculty in these discussions would presumably also be of value.

Schedule

Taken together, the above elements define the course schedule. An initial meeting is needed to introduce the topics of the course as well as how the course will run. In practice, two hours is about right for an introductory lecture characterized by interaction and engagement. This is the only lecture given by the course instructor, defined by a PowerPoint presentation of her or his own choosing. There is no course textbook or assigned readings other than options that might come up based on plans for or discussion following a particular student presentation. If the class is to be divided into ten groups, each of which is responsible for a presentation of approximately one hour, then an additional five sessions of two hours are needed. Finally, an additional two hours is needed for a faculty panel with 3 or 4 faculty answering a reasonable sampling of questions raised by the students. In short, the class in the incarnation being described here meets seven times for two hours each (Table 1)

Table 1.

Course schedule. The course meets seven times for two hours each as shown below.

Meeting	Description
1	Introduction and overview of the course
2–6	Two student group presentations of ~one hour each
7	Faculty panel

Student Engagement

In its current form, this course is distinguished by its student-centered approach. Classes are small, making it easier to focus not on content delivery, but on engaging with pressing issues in science and technology. Participants take the lead in presenting on topics they choose and moderating the discussion (with instructor input as needed). In practice, students chose the focus of the course, the discovery of issues that they, as scientists, find concerning, and have the space and time to think about their responsibilities as scientists and as citizens who have areas of scientific expertise. It is presumed that the course objectives will be best met when students discover that they are concerned, worried, or moved to action rather than when we tell them they should be more ethical, more responsible, or more worried about these issues. This can be accomplished by asking questions, as needed, to challenge the students to generate a richer approach to ethical considerations. For example: What might be the consequences of these actions? Who might be harmed? What are the options? What are the pros and cons of the different choices? What would you do? Why? What is your experience? What could you or others do to decrease the risk of problems in the future?

Evaluations

Early attempts at course evaluation were largely focused on evaluating the strengths and weakness of the course instructor (e.g., “using a numerical scale, with 1=poor and 5=excellent, how would you rank this instructor’s ability as a teacher?”). In the following years, based in part on anecdotal discussions with students, on hearing from people with expertise in effective teaching, and on some of our own qualitative research regarding the impact of RCR courses, we attempted to get a richer view of perceptions by asking more open-ended questions such as “What changes, if any, has this course made in your confidence or ability to handle situations involving possible scientific misconduct?” or even more open-ended: “How if at all have your awareness, knowledge, skills, or attitudes been changed by participating in this course?” In later years, as we reflected more on what we heard from the students on these evaluation forms as well as anecdotally, what we hoped to accomplish with these courses, and what was realistic, we developed questions about the extent to which students found themselves discussing course topics outside of class. We would argue that this simple endpoint, encompassing the interest, will, and ability to talk about ethical issues outside of the classroom, is more important than almost any other outcome (e.g., knowledge about the rules or increased skill to make ethical decisions). Unless someone is open to the possibility of an ethical issue to be addressed, excellent knowledge and skills will not be useful.

Course Impact

As for nearly all such courses, meaningful assessment of outcomes remains an elusive target (Mumford, 2017). However, we currently use four indications that collectively suggest a positive impact of this course format. First, the instructors of these courses at UC San Diego have taught and observed numerous similar courses locally, nationally, and internationally. Our collective impressions are that the course meets our learning objectives (noted above), that it does so as well or better than any other courses we have seen, and that it is more effective than most. Second, student evaluations of the instructors and quality of instruction are very high (Table 2: Instructor). Third, student self-assessments of success in achieving the stated learning objectives is also very high (Table 2: Objectives). Finally, one limited measure of the extent to which the course is having an impact on these students is whether they find the material sufficiently engaging to continue conversations outside of class with others. On that count, the students self-report a very high rate of such conversations: Students on average report having talked outside of class about what they heard in the course for an average of nearly half (2.8) of the six meetings preceding the final meeting of the course (Table 2: Further Discussions).

Table 2.

Average anonymous student responses to selected course evaluation questions. Questions about the Instructor and Objectives were rated on a 5-point Likert scale (1=strongly disagree, 5=strongly agree). The Further Discussions questions are self-reported number of meetings (0–6) that resulted in discussion outside of class. Data represent 95% confidence intervals for average evaluations for each of ten sequential iterations of the course given by seven different instructors from Spring quarter 2019 through Winter 2020.

Instructor: The instructor of this section of the course was effective overall.	4.7 ± 0.26
Objectives: This course helped me to meet the stated objectives, which include…
1. know rules, issues, options, and resources for research ethics	4.3 ± 0.15
2. understand the purpose and value of ethical decision- making	4.6 ± 0.15
3. have a positive disposition toward research ethics	4.6 ± 0.23
Further Discussions: Of the previous six meetings of this course, how many (0–6) resulted in you having further discussions outside of class?	2.8 ± 0.23

Adaptability

While clearly this approach is not the only way to teach about RCR, our experience suggests that student engagement is high, students and faculty rate the course highly compared to other courses they have seen, students are sufficiently motivated by course content to initiate further discussion outside the classroom, and the model is easy to implement. Faculty new to RCR, including early career investigators, report having successfully adopted the course despite lacking previous experience with such teaching, and one of us (DKP) developed an equally successful adaptation of the course with variations on a number of components including number and duration of sessions and inclusion of students from the humanities and social sciences.

Best Practices

The efforts described in this manuscript were carried out under the assumption that RCR training should first be about fostering a culture of integrity, not about merely satisfying requirements to receive federal funding. Therefore, best practices for such training should presumably be characterized by a focus on both the desired outcome (raising awareness and attention to responsible research) and the best methods for achieving educational outcomes based on what we know from scholarship on teaching and learning. Taken together, this iterative approach to developing our institution’s course should at the very least be considered a minimal practice for how RCR training should be developed and carried out in research institutions and organizations. And although we are reluctant to define our particular course as a “best” practice, it seems it is at least one way to meet the letter and spirit of requirements for such training.

Research Agenda

The premise of our approach to RCR education can be summarized as a focus more on institutional rather than individual outcomes. Successful completion of our course is based on engagement and participation in discussion with others in the class, and a primary outcome was (self-reported) conversations with others outside of the class. This is distinguished from other kinds of teaching and learning that emphasize assessment of what knowledge or skills an individual has learned. Admittedly, the merits of our framing can be debated, but it also suggests two kinds of research worth pursuing. The first is more theoretical and abstract: Can and should RCR teaching focus on institutional, collective outcomes rather than individual outcomes? The second is empirical: Do courses such as the one described here have a positive impact on institutional culture? Unfortunately, this empirical question is not easily studied as it is neither clear what would be a robust measure of success nor how one can readily separate out the impact of a given course from the many other factors that likely have an impact on the willingness of researchers to identify, discuss, and address the ethical challenges of their scholarship.

Educational Implications

The primary, relevant audience for this article consists of those who teach or who are tasked with teaching about the responsible conduct of research. Nominally, these stakeholders might recognize opportunities for strengthening their teaching simply by reviewing this manuscript. However, a more intentional approach to sharing this model might be accomplished through train-the-trainer programs such as the Responsible Conduct of Research Instruction Workshops sponsored by the US Office of Research Integrity in 2017 (San Diego), 2018 (Atlanta), and 2019 (Salt Lake City).