Teaching and Assessing the Responsible Conduct of Research: A Delphi Consensus Panel Report

Abstract

In an effort to foster research integrity, the National Institutes of Health and the National Science Foundation mandate education of all trainees in the responsible conduct of research (RCR). Nevertheless, recent studies suggest that rates of questionable research practices and scientific misconduct are both high and considerably underreported. In part, this may be due to the fact that some ethical norms (e.g., authorship assignment) are far from clear and researchers are unsure how to respond to perceived misconduct. With funding from the U.S. Office of Research Integrity (ORI), we convened four panels of experts to develop a consensus on the overarching goals and teaching content of RCR instruction. Our panelists recommended nine overarching objectives for RCR instruction that require us to rethink common modes of instruction, and they identified issues and standards that should be covered within controversial areas such as authorship assignment and whistle-blowing. Additionally, our experts recommended two new core areas for RCR instruction: The social responsibilities of scientists and current topics in RCR.

Introduction

Responsible Conduct of Research (RCR) education and training too often emphasize rules like “Do not falsify data” or “Do not plagiarize.” These are simple extrapolations of what most researchers learned in kindergarten: lying and stealing are wrong. Reminding researchers of such rules involves stating the obvious, with the result that RCR education and training may be perceived as boring, unnecessary, and ineffective.

However, not all issues in research ethics are so clear-cut. In a survey by Martinson, Anderson, & de Vries (2005) of over 1,700 researchers, 33% reported engaging in so-called “questionable research practices” such as dropping data points from analyses based on a hunch or inappropriately assigning authorship. The example of inappropriate authorship is particularly instructive. First, practices for assigning authorship vary across disciplines (Steneck, 2004). Second, even in a discipline such as medicine, in which international standards have been published (International Committee of Medical Journal Editors, 2007), authorship assignment has not become standardized. A recent review of 234 biomedical journals found that 41% gave no guidance about authorship and only 19% were based on the current criteria of the International Committee of Medical Journal Editors (Wager, 2007). Uncertainty about criteria helps to explain the high rates at which researchers admit to assigning authorship in a questionable manner. Yet, given a lack of standardized criteria within professions, even RCR instructors are uncertain what should be taught in the area of authorship.

While rates of strict research misconduct (data falsification, fabrication, or plagiarism) are much lower than rates of questionable practices, they are also higher than many might assume. A survey by the U.S. Office of Research Integrity (ORI) of researchers holding funding from the National Institutes of Health (NIH) at 605 different institutions, inquired into the number of times researchers had observed suspected research misconduct in their own departments over the previous three academic years (Titus, Wells, & Rhoades, 2008). A total of 2,212 researchers completed the survey (yielding a 51% response rate); they reported observing a total of 201 instances. By extrapolating this rate of observed suspected misconduct —assuming that the 49% who did not respond observed no instances of misconduct — the authors estimated that there are more than 2,300 observations of likely misconduct per year in research funded by the U.S. Department of Health and Human Services (DHHS).

Given that ORI receives an average of only 24 institutional investigation reports per year (approximately 1% of the estimated incidences observed), these numbers suggest the need for RCR education and training — not only to reduce rates of misconduct, but also to provide guidance to researchers in how to respond to observed misconduct. Yet this topic is also controversial. Real-world decisions regarding whistle-blowing are often far more complex (Smith, 2006) and their consequences far more devastating (Couzin, 2006) than ethics textbooks suggest. While it is not sufficient for RCR instructors to remind people of a duty to report misconduct, it is unclear precisely what content or standards should be taught.

In 2000, ORI identified nine cores areas that RCR courses should address: (1) data acquisition, management, sharing, and ownership; (2) mentor/trainee responsibilities; (3) publication practices and responsible authorship; (4) peer review; (5) collaborative science; (6) human subjects; (7) research involving animals; (8) research misconduct; and (9) conflict of interest and commitment. While these core areas provide a useful initial framework, there is no evidence of professional consensus that ORI’s list includes the most important areas of RCR, nor what content should be taught and assessed within the core areas (Steneck & Bulger, 2007). For example, Pimple (2002) has recommended approaching RCR through the lens of six domains, some of which overlap with the nine core areas, and some of which extend into new areas such as social responsibilities (including fiscal responsibilities, advocacy by researchers, and environmental impact).

RCR trainers may also have different goals in mind: to convey knowledge of right and wrong; to foster professional virtues; to inculcate values that support good science, to raise awareness of ethical issues; to motivate people to do what is right; and — most ambitiously — to improve behavior (DuBois, Ciesla, & Voss, 2001). The behavioral goal is probably the most widely proffered — even if controversial — insofar as ethics instructors frequently begin courses, textbooks, or funding proposals by citing instances of scientific misbehavior, thus implying that RCR training can help prevent such events. In this vein, one leading research administrator writes, “the value of … RCR education from an administrative perspective can be summed up in the oft-used adage, an ounce of prevention is worth of pound of cure” (Vasgird, 2007, p. 835).

Two studies examined the content and goals of RCR education and training. In 2005, Heitman and Bulger published a content analysis of 20 RCR textbooks. Content reflected each of ORI’s core areas and became more comprehensive after ORI published its policy on RCR instruction in 2000. The authors also identified gaps in the core areas of compliance, ethics of lab safety, institutional responsibilities, and the role of scientists in society (Heitman & Bulger, 2005). Kalichman and Plemmons (2007) studied the goals of existing education and training programs. They conducted interviews with 50 instructors and identified over 50 distinct goals pertaining to knowledge, skills, attitudes and behavior. They found that actual educational goals varied widely across instructors.

These two studies reinforce the need to pursue consensus on RCR instruction. On the one hand, important gaps appear to exist in RCR textbooks (e.g., institutional responsibilities and the role of scientists in society), while on the other, the study of actual education and training programs identified over 50 distinct educational goals, which varied widely across instructors. Add this to the vagaries surrounding authorship and whistle-blowing, and a muddy picture of the goals and content of RCR instruction emerges.

Whereas these previous studies examined the goals or content of existing RCR education and training programs and materials, our project sought to establish a consensus among experts on what RCR education and training should look like. We addressed four specific questions:

1. What should be the overarching goals of RCR training (e.g., knowledge, problem-solving skills, or virtue)?

2. Are the nine core areas of RCR instruction identified by ORI complete, or should additional core areas be addressed?

3. Within the core areas, what specific content should be taught?

4. What objectives and content should be assessed?

Methods: Delphi Expert Panels

About Delphi Consensus Panels

One way of developing recommendations for a field is to convene a diverse panel of experts to engage significant questions. Such an approach is regularly used by the U.S. National Academies of Science to address questions in the fields of engineering, medicine, and science. With funding from ORI we used an online Delphi panel process to foster an expert consensus. Delphi panels involve administering a questionnaire to groups of individuals across several rounds with the aim of identifying shared evaluations or recommendations (Ferguson, 2000). Key elements of the Delphi process are a structured flow of information, controlled feedback to participants, statistical analysis of responses, and participant anonymity. Interactions among panel members are controlled by a coordinator, who filters feedback and organizes data for subsequent presentation in the next round. The Delphi method maximizes the benefits of group decision-making while the anonymity of the process minimizes limitations such as domineering group members, personality conflicts, or groupthink (Delbecq, Van de Ven, & Gustafson, 1975). Other advantages to an online Delphi method include its relatively inexpensive cost and convenience for participants, who can access the survey at any time of day.

Delphi Panel Procedures

Because few people possess expertise in all areas of RCR, we formed four separate expert panels. Each panel worked independently and simultaneously. Our Delphi process involved multiple rounds of questioning. Round 1 consisted of an open-response format. Panelists were directed to one of four websites corresponding to their panel assignment(s), where responses were collected in text-boxes.

Our Objectives panelists were asked: (1) What should be the overarching educational objectives of RCR instruction; and (2) Are the nine core areas of RCR instruction complete, or should new core areas be addressed within RCR instruction?

Scientific Data panelists were asked: Within RCR instructional programs, what specific topics should be taught and assessed in the areas of: (1) Data acquisition, management, sharing and ownership; and (2) Research misconduct?

Scientific Relationships panelists were asked: Within RCR instructional programs, what specific topics should be taught and assessed in the core areas of: (1) Mentor/trainee responsibilities; (2) Collaborative science; and (3) Conflicts of interest and commitment?

Scientific Publications panelists were asked: Within RCR instructional programs, what specific topics should be taught and assessed in the core areas of: (1) Publication practices and responsible authorship; and (2) Peer review?

We excluded from our project two of ORI’s nine core areas for RCR instruction: human subjects and animals. There were several reasons for this: (1) Institutional Review Boards and Institutional Animal Care and Use Committees typically mandate ethics training that is separate from general RCR training; (2) the scope of these core areas is very large (thus excluding them made the project more manageable); (3) it appears that significant consensus exists on what needs to be covered in such courses; and (4) these areas do not comprise ORI’s primary areas of focus for education and oversight.

After all participants had completed round 1, their responses were carefully condensed, re-worded and organized into topics and subtopics to enhance clarity and prevent redundancy. Round 2 involved presenting panelists with the revised lists of topics they had generated and asking them to rate on a four-point scale the importance of teaching each topic in an RCR course: 1 = Unimportant, 2 = Less important, 3 = Important, and 4 = Very important. Panelists were also asked to make additional comments about the wording or clarity of each item.

Topics receiving a vote of “Important” or “Very Important” from at least two-thirds of panelists were deemed to meet consensus criteria and were presented to panelists in the next round, after they were revised according to the panelists’ comments. Topics not meeting consensus are displayed in the tables below, with their corresponding consensus values and mean scores.

Round 3 added to Round 2 by re-asking panelists the importance of teaching each item, and also asked panelists to rate the importance of assessing each item within an RCR course. Assessment rankings followed the same four-point scale used in the previous round. We asked separately about the importance of teaching and of assessing goals and content because we believed instructors might think some material is worth teaching without the need to assess learning (e.g., historical cases taught simply to provide context).

In each round, we asked panelists to prescind from whether it is feasible to assess a goal or topic and to focus on the importance alone.

Recruitment

We recruited experts for our panels during October and November 2006. The three rounds were conducted from November 2006 through June 2007. Recruitment began with: (1) a literature search to identify authors actively researching and publishing in RCR; (2) a review of ORI Annual Reports from 2000 through 2005 to identify those who received ORI contracts and grants; and (3) a review of recent research administrative and RCR conference programs to identify those who had presented on relevant topics. Based on these activities we generated a list of experts with overarching knowledge of RCR, many of whom provide RCR training.

Additionally, we produced a list of Chief Research Officers, scientific journal editors, and pre- and post-doctoral trainees who we believed would be interested in participating in our consensus project. From the resulting list of possible panelists, the Project Director, in consultation with ORI, selected those who were both qualified to serve on a particular panel and who represented diverse backgrounds. Recruitment letters were sent to these individuals, asking them to volunteer without compensation for a total of 1.5 hours (30 minutes for each round) per panel, over approximately nine months. Those who declined participation, but represented a subgroup of interest, were asked to provide a recommendation for another possible participant.

Overall, 41 individuals served as panelists on either one or two panels, participating in at least two of three rounds on any particular panel. (Individuals selected to serve on the Objectives panel were also asked to serve on one of the remaining three panels — Scientific Data, Scientific Relationships, and Scientific Publications.) The project retained nearly all experts across three rounds of questionnaires. The Objectives panel had 18 total panelists, with 16 to 18 panelists participating in any given round. The Scientific Data panel had 13 panelists with 12 to 13 participating per round; Scientific Relationships had 14 panelists with 12 to 14 participating per round; and Scientific Publications had 13 panelists with 10 to 12 participating per round.

Compliance

This project was presented to the Institutional Review Board at Saint Louis University, which concurred with the project director that the project did not constitute research because it was aimed at producing a consensus among experts rather than generalizable knowledge. The participants were experts serving on a panel seeking consensus on recommendations (akin to serving on a committee of the National Academies of Sciences).

Results

We defined a consensus as two-thirds of panelists supporting a rating of important or very important. What follows are highlights of our findings.

New Core Areas

The Objectives panel initially proposed several possible new core areas of RCR instruction. These are listed in Table 1. The panel reached a consensus on two new core areas: Social Responsibilities of Researchers and Current Issues in RCR. Because consensus on these new core areas only emerged through the process itself, the panelists were unable to propose content for these new areas, as they did with the pre-existing ORI-recommended core areas. However, some of our expert panelists have independently described in publications possible content that could be covered under such headings (Bulger & Heitman, 2007; Kalichman, 2002; Pimple, 2002). For example, under “social responsibilities,” Pimple (2002) includes research priorities, fiscal responsibilities, public service, public education, advocacy by researchers, environmental impact, and forbidden knowledge. Under current topics, a wide variety of issues might be discussed. For example, in today’s environment, instructors might want to discuss NIH policy on stem cells or the relationship between science and politics.

Table 1.

Proposed Additional Core Areas

Core Area	Percentage of panelists rating item as “important” or “very important” (Mean score)
	Teaching	Assessing
1. The financial and operational responsibilities of Principal  Investigators	50 (2.56)	N/A†
2. Social responsibilities of researchers	89* (3.28)	47 (2.65)
3. Historical background in responsible conduct of research	61 (2.72)	N/A†
4. Current issues in responsible conduct of research	89* (3.28)	50 (2.61)
5. Lab safety and environmental health	56 (2.72)	N/A†
6. Philosophy of science, including roles of bias and world  views in science	39 (2.50)	N/A†

Legend:

^*Item achieved a “consensus” by receiving a rating of important or very important from two-thirds of panelists

^†Not applicable because these items were eliminated after round 2 and their importance of being assessed was not measured

Objectives of RCR Training

Table 2 presents nine overarching objectives that the panel agreed should be taught in RCR training programs. Panelists supported assessing four of these nine objectives: fostering understanding of the importance of RCR and the consequences of misbehavior; examining how ethics may go beyond compliance with regulations; fostering sensitivity to ethical issues; and developing ethical problem-solving skills.

Table 2.

Overarching Educational Objectives for RCR Instruction

Topics (Subtopics indented)	Percentage of panelists rating item as “important” or “very important” (Mean score)
	Teaching	Assessing
1. Understand the importance of RCR	94* (3.59)	81* (2.94)
a. Know the history of research, including historical examples of    research misconduct and unethical conduct	82* (3.29)	56 (2.50)
b. Understand the social context of research	94* (3.29)	56 (2.78)
c. Consider consequences of unethical conduct in research for self,    institution, science, and society	100* (3.83)	89* (3.22)
2. Identify sources of RCR regulations and policies	83* (3.22)	50 (2.67)
a. Federal regulations	83* (3.22)	61 (2.78)
b. State laws	47 (2.65)	N/A†
c. Institutional policies	78* (3.17)	61 (2.83)
3. Examine limitations of RCR regulations and policies and variations in  standards across fields, institutions, and labs	83* (3.17)	61 (2.67)
a. Understand that regulations permit discretion and creative problem    solving	89* (3.28)	61 (2.72)
b. Understand that regulations require discretion and creative problem    solving	61 (2.94)	N/A†
c. Understand that regulations do not cover all ethical responsibilities	94* (3.50)	78* (3.11)
4. Understand key distinctions within the field of RCR	89* (3.22)	72* (2.83)
a. Distinctions within ethics, such as ethically obligatory, prohibited,    and praiseworthy actions	47 (2.65)	N/A†
b. Distinction between ethical and regulatory duties	83* (3.33)	56 (2.67)
c. Distinction between research misconduct and questionable research    practices	83* (3.33)	67* (3.06)
5. Foster research integrity or professional character	94* (3.65)	44 (2.61)
a. Motivate morally good action	83* (3.22)	39 (2.56)
b. Inculcate professional values such as pursuit of truth, honesty,    intellectual humility	100* (3.39)	50 (2.72)
6. Foster ethical sensitivity or the ability to identify ethical issues in the  conduct of research	94* (3.50)	83* (3.06)
a. Identify common ethical issues such as those addressed within the    core areas of RCR	83* (3.06)	72* (2.89)
b. Identify threats to RCR, including pressures in the research    institution and personal bias	100* (3.67)	61 (2.83)
c. Distinguish between ethical responsibilities in research versus other    professional activities such as education or clinical care	50 (2.50)	N/A†
7. Develop ethical problem-solving skills	89* (3.44)	78* (3.11)
a. Knowledge of relevant ethical frameworks, theories or principles	67* (2.72)	33 (2.39)
b. Ability to identify key elements of an ethical situation, including    stakeholders, relevant ethical and legal norms, relevant facts, and    options	89* (3.39)	67* (2.89)
c. Ability to critically reason using ethical principles or values	83* (3.39)	78* (2.94)
d. Ability to identify ethical resources, such as mentors, peers,    institutional officers, educational resources, and consultation services	83* (3.28)	56 (2.72)
8. Examine ways of preventing ethical problems in research	89* (3.22)	61 (2.67)
9. Provide an open forum for discussion of individual RCR concerns and  challenges	83* (3.33)	50 (2.33)

Legend:

^*Item achieved a “consensus” by receiving a rating of important or very important from two-thirds of panelists

^†Not applicable because these items were eliminated after round 2 and their importance of being assessed was not measured

Instructional Content

Within the seven core areas of RCR instruction that we examined, the panels achieved a consensus on the importance of teaching 43 main topics (with 0–6 specifications of content within each main topic). They supported assessing learning in 21 of these 43 main topics.

Tables 3–9 provide results from the three panels dedicated to ORI’s core areas of RCR instruction (See Appendix A). To illustrate the topics identified, within the core area of publication practices and responsible authorship, the panelists identified nine main topics instructors should address: the significance of authorship; authorship assignment; inappropriate authorship practices; dealing with controversies surrounding authorship; scientific responsibilities of authors; poor publication practices; protecting privacy in publications; addressing the study’s ethical compliance within articles; and responsible disclosure of scientific information within the popular press. Within most of these areas, further subtopics were recommended. For example, under the controversial topic of “authorship assignment” our panel arrived at a consensus that RCR courses should address: criteria for authorship (including substantial intellectual contribution to the study or paper and familiarity with and approval of the final text); the ideal of transparent contributions (i.e., descriptions of authors’ roles on the paper); how to deal with multiple authors; the appropriateness of discussing authorship at the outset of a project; the purpose and examples of acknowledgements versus authorship; and discussion of variations of published standards and norms across disciplines. Similarly, within the general topic of “responding to misconduct,” panelists identified several specific issues that should be addressed: the responsibilities of whistleblowers; the evidence needed to report misconduct; protections for whistleblowers; and alternatives to whistle-blowing, including illustrations of good and poor responses to observed misconduct.

Conclusions

Based on our review of the literature, our project appears to be the first attempt to convene a large group of experts to determine the ideal objectives and content of RCR instruction and assessment.

The project is limited in that it reflects the consensus among specific individuals; were different individuals selected, our consensus would likely be different. Moreover, we asked experts to consider RCR instruction in general — regardless of the trainee populations. If given the opportunity, experts might recommend different educational objectives or topics for undergraduate science students versus independent investigators.

Despite its limitations, our project reflects the consensus of individuals with considerable expertise, and ORI is exploring ways to disseminate our findings and recommendations to RCR instructors. We believe our results may guide the development and quality improvement of RCR education and training programs in several ways.

First, our results provide instructors with guidance in developing content for RCR curricula. For example, our project identified issues and standards that should be addressed across disciplines within controversial areas such as authorship attribution and whistle-blowing. This may help investigators who feel uncertainty regarding what to teach in the current absence (described above) of clear professional standards in some areas like authorship. Certainly the nine major topics recommended by the panel on publication practices would provide a useful starting point. Moreover, they may empower authors themselves in dealing with others on matters of authorship and acknowledgements, particularly in interdisciplinary research, where standards may vary.

Our experts further proposed two new core areas for RCR instruction: social responsibilities of researchers and current issues in RCR. Insofar as some popular training programs have limited their treatment of topics to the nine core areas originally proposed by ORI in 2000, this development may encourage the teaching of a variety of new topics such as research priorities, fiscal responsibilities, advocacy by researchers, or the relationship between science and politics.

Second, our panels identified important knowledge that should be assessed. Student performance on corresponding test items might provide an important measure of how well a course or training program fosters relevant knowledge and concepts.

Third, and most importantly, our project produced a list of objectives for RCR instruction that transcends the rote knowledge of regulations and basic societal expectations. For example, our panel believed that RCR instruction and education should foster integrity or professional character, examine how ethics may exceed compliance with regulations, and develop ethical problem-solving skills. These objectives may be described as promoting the development of researchers as individuals of integrity, ultimately contributing to the creation of a culture of ethics and integrity that goes beyond minimum compliance or risk management. Consider, for example, the matter of informed consent. Currently, no regulations require formal assessment of the decisional capacity of potential research participants. Yet arguably, an investigator of integrity who seeks to transcend mere compliance will recognize the need to ensure that participants understand consent information and will have the problem-solving skills to identify what options exist for assessing capacity and to recognize which options best fit particular circumstances.

These more robust objectives may have far-reaching consequences for how RCR training is provided. As Kalichman (2007) observes, “active learning modalities are more effective than passive learning … We are more likely to internalize and understand new information when challenged to do something with it than when someone simply tells us what we ‘should’ know”(p. 872). Yet formal RCR instruction is often restricted to passive online reading or lecture formats. While such instructional formats may foster rote knowledge, we question whether they are well suited to fostering professional character, ethical problem-solving skills, and other higher-order objectives. In contrast, instructional methods that provide a framework for reasoning through complex ethical dilemmas — particularly dilemmas or cases that are relevant to day-to-day work—are most effective in fostering moral reasoning and ethical decision-making skills (Bebeau, 1995; Kligyte, Marcy, Sevier, Godfrey, & Mumford, 2008).

Given the extensiveness of the content and the complexity of the objectives our experts recommended for RCR instruction, it is unlikely that any single education or training intervention will meet all of the goals. We recommend that, in addition to offering generic instruction on RCR aimed at knowledge of many topics, institutions develop education and training programs tailored to the population they serve (Heitman & Bulger, 2005; Kalichman, 2007). The Council of Graduate Schools’ (2006) recent report on Graduate Education for the Responsible Conduct of Research explicitly recommended such a two-tiered approach to RCR instruction, noting that “graduate students respond best to RCR training that is directly relevant to their experience as graduate students” (p. 25). Such specialized courses might be less comprehensive even as they are more relevant and engaging, perhaps focusing more on the development of high-order skills as described above. Most importantly, formal RCR instruction should be only one component in the overall project of fostering research integrity. Other components include: mentoring; the institutional climate; the establishment, communication, and enforcement of clear policies by institutions, funding agencies, and journal editors; and codes of ethics from professional societies (Adams & Pimple, 2005; Institute of Medicine & National Research Council, 2002; Macrina, 2007). In combination with formal RCR training, such efforts might eventually achieve some of the loftier goals our panels set for the field of science.

Appendix A

Tables on Proposed Content for Core Areas of RCR Instruction

Table 3.

Proposed Content for “Data Acquisition, Management, Sharing and Ownership”

Topic (Subtopics indented)	Percentage of panelists rating item as “important” or “very important” (Mean score)
	Teaching	Assessing
1. Ethical values behind the scientific standards for data acquisition,  management, sharing, and ownership	92* (3.58)	75* (2.83)
a. Confidentiality and privacy	100* (3.67)	92* (3.08)
b. Trustworthiness, honesty, and transparency	100* (3.75)	67* (2.92)
c. Right to property or to prosper from work	58 (2.67)	N/A=
d. Scientific collegiality and virtue of sharing	100* (3.50)	67* (2.75)
e. Value of having regulations and standards	75* (3.25)	58 (2.75)
2. Variations in lab practices—legitimate and illegitimate variations	92* (3.42)	58 (2.83)
3. Data acquisition issues	100* (3.82)	82* (3.27)
a. Informed consent or permission to gather or use data	100* (3.83)	83* (3.42)
b. Sampling and data selection	100* (3.75)	83* (3.33)
c. Verifying and cleaning data	100* (3.67)	75* (3.17)
4. Data storage, protection, and archiving	92* (3.50)	67* (2.92)
a. Techniques for entering, storing, and archiving data	64 (2.82)	N/A=
a. Data storage longevity (how long to save data and what    format)	83* (3.17)	58 (2.67)
b. Data protection and backup	92* (3.25)	67* (2.83)
c. Unique issues pertaining to special kinds of data, such as    tissue, DNA, photographic data	92* (3.33)	50 (2.83)
5. Data Sharing	100* (3.50)	67* (2.92)
a. How and when data should be shared, advantages and    disadvantages	100* (3.50)	75* (2.83)
b. Transferring data	64 (2.55)	N/A=
c. Acceptable and unacceptable uses for shared data	100* (3.45)	82* (3.00)
6. Legal aspects of data ownership and rights	92* (3.58)	83* (3.25)
a. Ownership of data, patents, copyrights, and intellectual    property	83* (3.50)	83* (3.08)
b. Institutional versus research rights to own and use data	92* (3.50)	75* (3.08)
c. Commercially useful data	100* (3.58)	75* (3.17)
d. Negotiating contracts	33 (2.50)	N/A=
7. Data privacy	100* (3.50)	67* (3.00)
a. HIPAA and other privacy rules	67* (3.50)	58 (2.83)
b. HIPAA and other privacy standards	55 (2.91)	50 (2.60)
c. Confidentiality protection techniques	100* (3.42)	75* (3.00)
8. Scientific methodology issues, including research design, objectivity,  and bias	92* (3.67)	92* (3.33)
a. Importance of research design	100* (3.75)	100* (3.50)
b. Elements of good scientific design and methodology	100* (3.75)	100* (3.42)
c. Proper use versus abuse of statistics	100* (3.75)	100* (3.45)
d. Challenges to maintaining objectivity in designing research    questions, controlling bias	92* (3.58)	92* (3.25)
9. Data reporting	100* (3.75)	83* (3.17)
a. Ethical issues when reporting data in publications	92* (3.67)	75* (3.08)
b. Responsibility to interpret findings appropriately to diverse    audience, scientific and otherwise	100* (3.58)	75* (2.83)
10. Special issues related to scientific roles	82* (3.18)	64 (2.73)
a. Obligations of students to supervise their own data collection    efforts	64 (2.91)	N/A=
b. Roles and relationships among team members	92* (3.25)	67* (2.58)
c. Who has the authority to make data related decisions	92* (3.25)	55 (2.55)

Legend:

^*Item achieved a “consensus” by receiving a rating of important or very important from two-thirds of panelists

^†Not applicable because these items were eliminated after round 2 and their importance of being assessed was not measured

Table 4.

Proposed Content for “Mentor/Trainee Responsibilities”

Topic (Subtopics indented)	Percentage of panelists rating item as “important” or “very important” (Mean score)
	Teaching	Assessing
1. Definitions and expectations of the mentor/trainee relationship	100* (3.75)	36 (2.36)
a. Defining research advisors, mentors, and trainees—across a variety of  settings including degree programs, postdoctoral training, and jobs	42 (2.50)	N/A†
b. Boundaries of the mentor/trainee relationship	100* (3.58)	45 (2.36)
2. Power relationships and the potential problems they involve	100* (3.58)	40 (2.50)
a. Power structures and hierarchical relationships within science and the  mentor-trainee relationship	92* (3.33)	25 (2.08)
b. Friendships and mentoring relationships	42 (2.50)	N/A†
c. Harassment, sexual and other types	67* (3.08)	42 (2.42)
3. Scientific responsibilities of the mentor	100* (3.42)	50 (2.58)
a. Promoting professional research skills, including identifying research  questions, writing proposals, conducting research, and publishing	92* (3.17)	33 (2.42)
b. Fostering research compliance (IRB, IUCUC, etc.), RCR, and integrity	100* (3.58)	75* (3.08)
c. Finding funding and negotiating grants and contracts	33 (2.25)	N/A†
d. Sharing discipline-specific wisdom on how to operate in the field	33 (2.33)	N/A†
4. Non-scientific responsibilities or roles of the mentor	67* (2.92)	42 (2.08)
a. Career counseling, including trainees with science and non-science career  goals	42 (2.42)	N/A†
b. Conflict resolution	67* (3.00)	25 (2.00)
c. Fostering autonomy with trainees while accomplishing mentor’s goals	67* (2.92)	25 (2.17)
d. Management skills	42 (2.50)	N/A†
5. Responsibilities of trainees within the mentor-trainee relationship	100* (3.42)	45 (2.45)
a. Work with integrity	100* (3.42)	42 (2.42)
b. Willingness to blow whistle or challenge misconduct and questionable  conduct	100* (3.50)	36 (2.36)
6. How to get the most out of the mentor/trainee experience	58 (2.67)	N/A†
a. Optimal characteristics of mentors and trainees	58 (2.58)	N/A†
b. Effective mentoring strategies and characteristics	83* (3.08)	27 (2.27)
c. Contracting for a good mentoring relationship	33 (2.33)	N/A†
7. Addressing challenges and problems in the mentor-trainee relationship	100* (3.25)	27 (2.27)
a. Conscientious refusal	58 (2.58)	8 (1.83)
b. Importance of clear communication of expectations	100* (3.25)	33 (2.42)
c. Dealing with diversity of cultures, races, and other personal traits	92* (3.25)	42 (2.33)

Legend:

^*Item achieved a “consensus” by receiving a rating of important or very important from two-thirds of panelists

^†Not applicable because these items were eliminated after round 2 and their importance of being assessed was not measured

Table 5.

Proposed Content for “Publication Practices and Responsible Authorship”

Topic (Subtopics indented)	Percentage of panelists rating item as “important” or “very important” (Mean score)
	Teaching	Assessing
1. The significance of authorship	91* (3.45)	55 (2.64)
a. The benefits of publishing	40 (2.70)	N/A†
b. The problems of inappropriate authorship for legitimate authors, illegitimate  authors, and science	91* (3.45)	73* (3.00)
2. Authorship assignment	91* (3.36)	64 (2.73)
a. Authorship criteria	91* (3.55)	64 (2.91)
i. Substantial intellectual contribution to study or text	100* (3.64)	73* (3.27)
ii. Familiarity with and approval of the final text	82* (3.36)	55 (2.91)
b. Ideal of transparent contributions	73* (3.00)	45 (2.45)
c. Multiple authors: how to determine senior/first author	82* (3.36)	55 (2.73)
d. Appropriateness of discussing authorship at outset of a project	91* (3.64)	64 (3.09)
e. Acknowledgments: purpose and examples (including faculty contributions to  students work)	90* (3.40)	60 (2.90)
f. Variation of standards and norms across disciplines	82* (3.00)	45 (2.27)
3. Inappropriate authorship practices	73* (3.36)	55 (3.00)
a. Ghost authorship	64 (3.09)	55 (2.73)
b. Forced or “courtesy” authorship, e.g., when students are asked to add  authors for political reasons	73* (3.27)	55 (2.82)
4. Dealing with controversies that arise in authorship	82* (3.36)	55 (2.73)
5. Scientific responsibilities of authors	91* (3.73)	91* (3.36)
a. Disclosure of funding sources and other sources of potential bias	100* (3.82)	82* (3.36)
b. Specification of any deviations from standard scientific practices	91* (3.55)	82* (3.27)
c. Full and accurate description of methods, procedures and analytic techniques  that allows repetition	91* (3.64)	82* (3.27)
d. Citation of relevant literature without bias	100* (3.55)	64 (3.00)
e. Duty to report findings accurately and completely, including reporting  critical or negative findings (even if they are contrary to own research  agenda)	100* (3.73)	82* (3.45)
6. Poor publication practices	91* (3.45)	73* (2.18)
a. Plagiarism versus proper citation or paraphrasing	100* (3.73)	82* (3.45)
b. Delay in reporting for commercial reasons	70* (2.80)	60 (2.60)
c. Publication bias	100* (3.36)	64 (2.82)
d. Text recycling; overlapping publication; duplicate and salami publication	100* (3.55)	64 (2.82)
e. Quality standards	91* (3.27)	64 (2.73)
7. Protecting privacy in publication	60 (3.00)	N/A†
8. Addressing compliance with ethical standards within articles (e.g.,  mentioning IRB or IACUC approval, and discussing ethically controversial  elements of a study)	100* (3.18)	55 (2.64)
9. Responsible disclosure of scientific information within the popular press	60 (2.60)	N/A†

Legend:

^*Item achieved a “consensus” by receiving a rating of important or very important from two-thirds of panelists

^†Not applicable because these items were eliminated after round 2 and their importance of being assessed was not measured

Table 6.

Proposed Content for “Peer Review”

Topic (Subtopics indented)	Percentage of panelists rating item as “important” or “very important” (Mean score)
	Teaching	Assessing
1. The significance of peer review	100* (3.64)	73 (3.09)
a. Peer review as a mechanism for quality assurance in publication and funding	100* (3.18)	55 (2.64)
b. The need for reviewers to be competent and genuine peers	91* (3.36)	64 (2.82)
2. Conflicts of Interest and Peer Reviews	100* (3.73)	91* (3.36)
a. Identifying potential conflict of interest reviewers may have	100* (3.73)	82* (3.18)
b. Managing conflicts of interest by excusing oneself from a review or disclosing and managing the conflict with the assistance of those directing the review	100* (3.82)	91* (3.27)
c. Other sources of peer review bias	82* (3.09)	55 (2.64)
3. Qualities of a good review/reviewer	82* (3.36)	55 (2.64)
a. Respecting confidentiality and intellectual property (e.g., by avoiding use of information and destroying manuscripts after review)	91* (3.27)	64 (2.91)
b. Fairness and objectivity	91* (3.55)	70* (3.10)
c. Collegiality—conveying a respectful and professional tone while offering critical feedback	80* (3.20)	40 (2.30)
d. Timeliness	82* (3.18)	45 (2.45)
e. Providing clear, scientifically competent, and complete reviews	91* (3.27)	64 (3.00)
4. Logistics of peer reviewing	50 (2.40)	N/A†
a. Format of written review	30 (2.20)	N/A†
b. Peer review process	60 (2.70)	N/A†
c. Selection of reviewers	50 (2.60)	N/A†
5. Responding to reviewers	82* (3.18)	60 (2.70)
a. Responding to competent reviews: the revision and resubmission process	60 (2.60)	N/A†
b. Responding to questionable, biased, or conflicted reviews: the roles of authors (PIs), editors, and scientific review chairs	91* (3.18)	64 (2.64)
c. Inappropriate responses to reviewers and modifications to publications or proposals	60 (2.50)	N/A†
6. Reviewer roles in ensuring RCR	82* (2.91)	36 (2.27)
7. Editorial responsibilities	55 (2.73)	36 (2.27)
a. Selecting appropriate reviewers	55 (2.73)	36 (2.27)
b. Attending to matters of RCR (proper authorship, disclosure of bias and conflicts, etc) – 2.70	60 (2.70)	N/A†
c. Respecting rights of rebuttal and mediating disputes	60 (2.60)	N/A†
d. Maintaining confidentiality	64 (3.00)	45 (2.55)

Legend:

^*Item achieved a “consensus” by receiving a rating of important or very important from two-thirds of panelists

^†Not applicable because these items were eliminated after round 2 and their importance of being assessed was not measured

Table 7.

Proposed Content for “Collaborative Science”

Topic (Subtopics indented)	Percentage of panelists rating item as “important” or “very important” (Mean score)
	Teaching	Assessing
1. The nature and advantages of successful collaborations	83* (3.17)	50 (2.50)
a. Reasons for collaborating	58 (2.83)	N/A†
b. Risks and benefits of collaborations	75* (3.08)	42 (2.17)
c. Identifying a good collaborator	83* (3.08)	33 (2.33)
2. Types of collaboration	63 (2.73)	22 (2.00)
a. Collaboration within an institution	67* (2.75)	17 (2.00)
b. Collaboration between institutions	58 (2.67)	8 (1.83)
c. International collaboration	58 (2.83)	N/A†
3. Working well with others	92* (3.25)	27 (2.27)
a. Identifying the authority and procedures for establishing collaborative  relationships	92* (3.00)	33 (2.25)
b. Defining and clarifying roles, responsibilities, and expectations in a  collaboration	100* (3.42)	33 (2.42)
c. Identifying mechanisms for ongoing decision-making	75* (2.92)	25 (2.17)
d. When are written agreements necessary, and what should be addressed in  contracts	92* (3.25)	75* (2.75)
e. Knowing how and when to end collaborative relationships	83* (3.00)	33 (2.17)
4. Dealing with challenges in collaborative relationships	100* (3.40)	40 (2.50)
a. Addressing failures in RCR or research integrity	83* (3.33)	82* (2.73)
b. Allocating rewards such as credit, authorship, ownership, and rights of use	100* (3.58)	83* (3.08)
c. Dealing with competition	50 (2.58)	N/A†
d. Addressing power discrepancies when junior scientists collaborate with  senior scientists	75* (3.00)	50 (2.58)
5. The role of institutions in collaborative science	58 (2.67)	N/A†
a. Working with appropriate officers	50 (2.58)	N/A†
b. Knowledge of institutional policies	83* (3.08)	50 (2.58)

Legend:

^*Item achieved a “consensus” by receiving a rating of important or very important from two-thirds of panelists

^†Not applicable because these items were eliminated after round 2 and their importance of being assessed was not measured

Table 8.

Proposed Content for “Research Misconduct”

Topic (Subtopics indented)	Percentage of panelists rating item as “important” or “very important” (Mean score)
	Teaching	Assessing
1. Significance of misconduct	100* (4.00)	100* (3.75)
a. History of scientific misconduct	82* (3.00)	42 (2.17)
b. Incidence rate of misconduct	58 (2.58)	N/A†
c. Consequences of misconduct for individuals, laboratories, science, and society	100* (3.64)	67* (3.00)
2. Factors that contribute to scientific misconduct	100* (3.73)	75* (3.25)
a. Effects of laboratory environment	100* (3.64)	75* (3.08)
b. Reward systems in academic and industry settings	100* (3.45)	67* (2.83)
3. Plagiarism	100* (3.91)	83* (3.33)
a. Definition and examples	100* (3.73)	92* (3.25)
b. Case studies with outcomes and punishments	83* (3.18)	58 (2.67)
4. Falsification	100* (4.00)	92* (3.50)
a. Definition and examples	100* (3.82)	92* (3.25)
b. Case studies with outcomes and punishments	100* (3.60)	73* (3.00)
5. Fabrication	100* (4.00)	92* (3.50)
a. Definition and examples	100* (3.82)	91* (3.18)
b. Case studies with outcomes and punishments	100* (3.55)	75* (3.00)
6. Other serious deviations from scientific best practices	80* (3.22)	60 (2.70)
a. Sabotage	58 (3.00)	N/A†
b. Questionable research practices (e.g., data manipulation)	100* (3.55)	75* (3.00)
c. Unintentional deviations	100* (3.45)	67* (2.92)
7. Regulations and policies addressing misconduct	82* (3.40)	82* (3.18)
a. The Office of Research Integrity’s role in addressing misconduct	92* (3.18)	50 (2.58)
b. Institutional policies	92* (3.36)	67* (2.92)
8. Responding to observed misconduct	100* (3.91)	92* (3.42)
a. Evidence needed to report misconduct	100* (3.73)	64 (3.00)
b. Whistle blowing, including responsibilities and protections for whistle blowers	100* (3.82)	92* (3.25)
c. Alternatives to whistle blowing with illustrations of good and bad responses	92* (3.45)	75* (2.92)
9. Studying taboo, controversial, or politically sensitive research topics	83* (3.09)	50 (2.50)

Legend:

^*Item achieved a “consensus” by receiving a rating of important or very important from two-thirds of panelists

^†Not applicable because these items were eliminated after round 2 and their importance of being assessed was not measured

Table 9.

Proposed Content for “Conflicts of Interest and Commitment”

Topic (Subtopics indented)	Percentage of panelists rating item as “important” or “very important” (Mean score)
	Teaching	Assessing
1. The significance of conflicts of interest	100* (3.73)	75* (2.83)
a. Historical examples of conflicts of interest in science	50 (2.58)	N/A†
b. Psychology and conflicts of interest, i.e., how conflicts of interest may cloud  judgment or influence decisions	83* (3.33)	50 (2.42)
c. The pervasiveness of conflicts of interest, including sponsored research	83* (3.25)	33 (2.17)
d. Consequences of conflicts for researchers, institutions, students and research  participants	92* (3.33)	33 (2.33)
e. Why conflicts of interest are pervasive and not always bad	83* (3.42)	50 (2.50)
2. Types, definitions, and examples of conflicts of interest	100* (3.55)	50 (2.50)
a. Financial conflicts of interest, including gifts and honoraria, patents, spin  off companies, SBIR/STTRs, personal investments, funding contracts with  industry	92* (3.58)	67* (3.00)
b. Non-financial conflicts of interest (e.g., recognition, publications,  promotions)	58 (2.67)	N/A†
c. Role conflicts (e.g., physician-researcher or teacher-researcher) and  conflicting duties to self, clients, institutions and society	75* (3.08)	50 (2.42)
d. Conflicts of interest are objective relationships—they do not imply actual or  intended wrong doing	83* (3.00)	33 (2.25)
3. Conflicts of commitment (i.e., dividing one’s percent effort within a job)—  definition, examples, and management	58 (2.67)	N/A†
a. Effort reporting rules	58 (2.67)	N/A†
b. Balancing sponsored research with other duties	58 (2.67)	N/A†
c. The perils of becoming over extended	67* (2.92)	33 (2.17)
4. Institutional conflicts of interest	50 (2.58)	N/A†
a. Conflicted oversight (e.g., IRB and IACUC members are employees who  review work of peers)	42 (2.50)	N/A†
b. Institutional investments and profits from research	42 (2.42)	N/A†
5. Managing conflicts of interest	100* (3.50)	67* (2.83)
a. Avoiding or eliminating conflicts of interest	100* (3.25)	58 (2.75)
b. Disclosing conflicts of interest / conflicts of interest and informed consent	100* (3.58)	83* (3.00)
c. Management plans, including, e.g., role separation	42 (2.67)	N/A†
6. Conflicts of interest law and policy	50 (2.58)	N/A†
a. Regulatory and statutory laws	50 (2.75)	N/A†
b. Institutional policies on conflicts of interest	82* (3.27)	55 (2.55)

Legend:

^*Item achieved a “consensus” by receiving a rating of important or very important from two-thirds of panelists

^†Not applicable because these items were eliminated after round 2 and their importance of being assessed was not measured