RESEARCH RECORDS AND THE RESOLUTION OF MISCONDUCT ALLEGATIONS AT RESEARCH UNIVERSITIES

Abstract

Accurate record keeping is an important part of the responsible conduct of research. However, there is very little empirical research on scientific record keeping. No one knows the incidence of serious problems with research records, the types of problems that occur, nor their consequences. In this study, we examined the role of research records in the resolution of misconduct allegations as a useful barometer for the incidence and types of problems that occur with records. We interviewed Research Integrity Officers (RIOs) at 90 major research universities and conducted focus groups with active research faculty. RIOs reported problems with research records in 38% of the 553 investigations they conducted. Severe problems with research records often prevented completion of investigations while problems that are more typical lengthened them by 2 to 3 weeks. Five types of poor record keeping practices accounted for 75% of the problems with incomplete/inadequate records being the most common (30%). The focus groups concurred with the findings from the interviews with RIOs, stressed the importance of the research group leader in setting and maintaining record practices, and offered additional insights. While university officials and faculty members have suspected for many years that there are serious problems with research record keeping, our study provides empirical evidence for this belief. By documenting some of the problems with record keeping in university-based research, the results of our study provide information that will be useful for policy development at academic institutions.

Introduction

Anecdotal evidence suggests that many cases of alleged scientific misconduct involve problems with record keeping (Office of Research Integrity, 2004; Shamoo and Resnik, 2003). In the most famous misconduct case in the past two decades, a federal appeals panel cleared Tufts immunologist Thereza Imanishi-Kari of 19 charges of scientific misconduct. The case, which focused on an article published in the journal Cell co-authored by Nobel Prize winner David Baltimore, lasted more than a decade. Though Imanishi-Kari denied that she had fabricated or falsified data, she did admit that she kept poor records. One wonders whether this case could have been resolved quickly if she had kept better records (Kaiser and Marshall, 1996; Kevles, 1998).

While inadequate record keeping may seem minor compared with falsifying data or ignoring human subject protections, there are many discussions of the importance of record keeping in papers, books, and documents on research integrity. Proper record keeping is “crucial to scientific research” (Macrina, 2000, p. 231) and “crucial to ensuring accountability in research” (Shamoo and Resnik, 2003, p. 30). The Panel on Scientific Responsibility and the Conduct of Research (1992), convened by the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine, cites the failure to maintain adequate research records as a questionable research practice.

While scientists recognize the importance of good record keeping, there have been few empirical studies of actual record keeping practices. No one knows how often poor record keeping leads to significant problems for researchers. Similarly, the types of poor record-keeping practices responsible for these problems need to be identified. Recently, Martinson et al., (2005) published a large study of questionable research practices. One of the 30 questionable practices was “inadequate record keeping.” Over one-quarter of the early and mid-career researchers funded by NIH reported that they had engaged in inadequate record keeping during the past 3 years.

If inadequate record keeping is common in scientific research, misconduct investigations should clearly reveal the problem. During a misconduct investigation, institutional officials need to be able to reconstruct all of the steps of the research process to determine the validity of the data. Investigators need to be able to determine who generated the data, when they generated the data, how they generated the data, what data they generated, where they stored the data, and why and how the data were processed or altered. Without complete and credible research records, it may be impossible to determine whether fabrication or falsification has occurred.

This study examined the role of good record keeping in the investigation and resolution of research misconduct cases. We began by interviewing research integrity officials (RIOs) responsible for investigating allegations of scientific misconduct at major research universities about their experiences examining research records during misconduct inquiries or investigations. Misconduct investigations are the only place we know of where external reviewers have access to the complete research records of projects that have been completed and usually published.

Materials and Methods

We obtained a list of Responsible Institutional Officials from the Office of Research Integrity, DHHS and matched the names with the top 160 research universities¹ based on R & D expenditures (2001 NSF Survey of R & D Expenditures). In some cases, different officials were responsible for overseeing research integrity cases in different parts of the university (e.g., medical schools often had a separate official).

Once the official with the formal responsibility of reporting misconduct investigations to the Office of Research Integrity was identified, that official was asked to identify the research integrity officials at the institution who had personal responsibility for performing misconduct investigations and/or inquiries. Sometimes these were the same officials, but often they were different. Once this person was located, a skilled telephone interviewer conducted a semi-structured interview during a prearranged appointment. The interview included a mix of short-answer questions (e.g., questions about incidence and severity) and open-ended questions about the nature of problems with research records. The interview often took over 30 minutes to complete.

In the summer of 2004, we conducted focus groups of active research faculty at 4 different research universities. These groups critiqued and expanded the results of our survey of RIOs. The participants also advised the research team on the best ways to approach future surveys of active researchers. Focus groups were held on the campuses of Duke University, the University of North Carolina at Chapel Hill, North Carolina State University, and East Carolina University.

Each focus group consisted of 12 faculty members who had recently conducted federally funded research. Their institution’s research integrity officer (see Acknowledgments) recruited the participants. Their research interests ranged from “basic” to “applied” and “clinical research.” Their academic disciplines included such diverse fields as agriculture, anthropology, biology, veterinary sciences, geology, and medicine.

Each focus group lasted 4 hours including a lunch during which the participants got to know each other. The participants were compensated for their time. Three or 4 members of the research team attended each focus group. After the sessions concluded, the research team discussed and recorded their observations. We videotaped each session and transcribed the tapes. Each member of the team reviewed the transcripts and suggested revisions to the initial findings. The results of the focus groups included in this article reflect the unanimous agreement of the research team.

Results

Ninety-six officials at 90 research universities completed the telephone interview. The university participation rate was 56% (90 universities participated out of 160 that were contacted). How did the universities who participated compare with those that declined? To answer this, we divided the population into 5 equal groups (32 universities in each group) based on their research funding. Using the same division points, we divided the final sample into 5 groups. The differences between the sample and the population are not statistically significant (chi square = 2.50, df = 4, p > .66). The most common reasons given to the interviewers for nonparticipation were being too busy, being unable to locate the appropriate person, and being new on the job.

About one-fourth (23%) of the 96 respondents reported that they knew of no misconduct cases at their universities. Another quarter of the respondents reported 1 or 2 cases (23%) or 3 to 5 cases (24%). About 14% reported 6 to 9 cases, and the final 17% reported 10 or more cases. The total number of cases reported was 553. The mean number of cases was 5.8 and the median was 3.0. The range was 0 to 50. This range is due, we believe, to the varying lengths of time the interviewed officials have been in their positions. Some had been in the position only 1–2 years while others had more than a decade of experience. As expected, the larger the research budget of the university, the more experience the RIO had handling inquiries and investigations. The Pearson’s correlation between the university’s research funding and the number of inquiries and investigations is .29 (N = 94, two-tailed significance = .005). This correlation is .25 (N = 94, two-tailed significance = .015)

Records in Misconduct Inquiries and Investigations

Table 1 presents information of the role of research records in the misconduct investigations conducted by the respondents.

TABLE 1.

Research records in misconduct investigations conducted by respondents²

	N Cases	N RIOs
Total Interviews		96
No Misconduct Cases Reported	0	23
Total Misconduct Cases	553	73
Cases Not Involving Research Records	171	48
Cases Involving Research Records	382	67
Cases with Research Record Problems	146	38

Over half (38/67) of the RIOs interviewed who had misconduct cases involving research records experienced cases with record problems. Such cases represented 38% of all cases that involve research records. In most of these cases, however, the quality of the records was mixed with some facilitating and some inhibiting the investigation. The RIOs reported that the records did not facilitate the investigation at all in only 2% of the problem cases.

We asked the RIOs to rate the significance of the “most serious,” “least serious,” and “typical problems” they encountered with research records during their misconduct investigations. To help assure a common rating scale for our diverse respondents, they were asked to rate the significance on a scale from 1 to 5 using the following definitions:

1. “slightly significant,” means there were difficulties that were easily overcome;

2. “somewhat significant,” means there were difficulties that were overcome with extra work;

3. “moderately significant,” means there were difficulties that were overcome with a lot of extra work;

4. “very significant,” means most difficulties were overcome with a lot of extra work but some issues were never resolved; and

5. “extremely significant,” means that many difficulties could not be overcome, making it impossible to use the research records to answer important questions for the inquiries or investigations.

The RIOs’ mean rating of their “typical problem (see Figure 1) was 2.5, indicating that all issues were resolved with “extra work” (2) or “a lot of extra work” (3). The respondents’ mean rating of their most serious problems was 4.0, indicating, “most difficulties were overcome with extra work but some issues were never resolved.” However, when we looked at the distribution of responses to these questions, it was interesting to note that 40% of the respondents rated their most serious problem as 5s (extremely significant) meaning that the problems were so serious the research records could not be used in the inquiry or investigation. In addition, almost 80% reported at least one case where the problems were so serious that some issues were never resolved and a few (10%) reported that the research records in their typical cases were so serious that some issues were never resolved.

FIGURE 1.

Mean rating of the significance of problems with research records encountered during misconduct investigations.

Respondents were also asked: “On the average, how many extra days were spent to resolve problems with research records?” The answers ranged from 1 to 60 days. The mean was 17.4 days and the median was 5.0 days.

Open-Ended Responses from Research Integrity Officials

To understand the kinds of research record keeping practices that created problems for the RIOs, we asked them to describe the most significant, next most significant, and any other significant problems that they had encountered with the research record during an inquiry or investigation. The categories of problems used in Table 2 were constructed through inspection of the responses. Statements made by respondents could be assigned to multiple categories since a single statement may include more than one type of problem (e.g., respondents refer to disorganized records that also contain incomplete documentation). Since a single answer could be assigned multiple codes, Table 2 reports more problems than the total number of respondents.

TABLE 2.

Problems with research records in misconduct investigations

Problem	N	%
Incomplete/inadequate documentation	41	29.9
Physical Access	19	13.9
Disorganized Records	18	13.1
Missing Records	17	12.4
Alterations/destroyed records	9	6.6
Volume of Records	5	3.6
Electronic Records-management	4	2.9
Electronic Records-obsolete software	3	2.2
Miscellaneous (11 problems)	21	15.4
Total responses	137	100.0%

The RIOs mentioned 5 types of problems much more frequently than other problems. Together, these represent almost 75% of the total. The first problem was much more than twice as common as any other problem. Almost a third (30%) of the responses mentioned incomplete or inadequately documented records. This problem arises when the records lack important facts (e.g., when the record was made, who made it; what experimental protocol was used, etc.) necessary to interpret the data or understand its relation to other data/research records. An inadequately documented record may have a great deal of information but certain key facts are missing so that an external expert cannot interpret the record.

The next 3 problems were mentioned with similar frequencies (12% to 14%). The second most common problem (14% of the responses) was gaining physical access to the research records irrespective of the form of the records, e.g., the records may be at another institution with a researcher who has left the university.

The third problem was disorganized records. This has 2 forms. First, the relationships among the records are difficult to understand, e.g., the chronology of the records may be unclear, or it may be unclear to which part of the project the records refer. Second, the records themselves may be physically scattered, e.g., the records may be in several, poorly labeled filing cabinets, in several different research notebooks, or as unorganized electronic files in several computer hard drives. The required data may also be so diluted by other information that makes retrieving the pertinent information a monumental task, e.g., when clinical researchers use patients’ medical records as their research records. This problem is related to the “incomplete/inadequate documentation” problem but differs in as much as the records seem to exist and be complete but a great deal of effort is required to organize them into a usable form.

The fourth problem was missing records. This problem occurs whenever big chunks of important data records describing experiments or studies are missing. Other records from the research project suggest that these records were produced but they cannot be located. Missing records differ from incompletely documented records in that in the former case the records were never created. The fifth problem was altered or destroyed records. Both the missing records and the fifth most frequent problem, altered or destroyed records, suggest the possibility of a deliberate effort to hide evidence or mislead investigators.

Focus Groups with Faculty Researchers

The focus group members readily accepted the results of the interviews with the university administrators as reflective of the types of problems present in academic research record keeping. They also identified additional types of poor record keeping practices, e.g., poor records of the deliberations/decisions reached among collaborators on a project. Such records are important when disputes among collaborators occur.

The focus groups discussed the possible causes of poor record keeping and the principles that promote good record keeping practices (Best Practices). They determined that many of the problems encountered by the RIOs were probably due to ineffective leadership by the mentor or research team leader, e.g., inadequate or improper training and oversight of students and subordinates by the research team leader. The focus groups emphasized the importance of the team leader or mentor to the integrity of the research process. In order to lead effectively, leaders must implement certain principles and practices. One principle is that leaders should set the standards (preferably in written form) for record keeping by specifying what information each group member should record in addition to the raw data itself. Another principle is that group leaders are responsible for providing (or arranging) the training and oversight of their personnel in record keeping. Such principles and practices would be applicable across scientific disciplines since they focus on research group management rather than the details of the scientific project.

The focus groups also identified principles and practices at the departmental or institutional level that would promote better record keeping, such as written guidelines and tools for record keeping, storage areas, and archiving services. Principles and practices for institutions and group leaders have been compiled into a ‘Best Practices’ document which has been presented in another article (Schreier et al., 2006).

As expected, the focus group members strongly supported norms of good record keeping as critical to good science. However, there was considerable concern that new burdensome record keeping requirements would be imposed by federal and/or other authorities without providing resources to cover the additional costs. The researchers also clearly understood that the research records belong to the university and were supportive of the institutional officials’ right to access the research records. The researchers noted that the fragmented nature of modern research, involving collaborations among colleagues in different departments and institutions, made immediate physical access to the complete research record difficult. They also noted that the volume and types of data collected, especially in clinical research, also make record keeping more difficult and less understandable³.

Combining the open-ended responses of the RIOs and the focus groups, we can see that the majority of problems center on 2 sets of issues: understandability of the records (e.g., incomplete/inadequate documentation, disorganized records) and access to the records (e.g., physical/electronic access, missing, incomplete, and destroyed records). Most of these problems seem to reflect carelessness, inadequate response to new circumstances (e.g., new equipment that records data in a new way) or lack of resources. The nature of modern research where rapid change in technical procedures and even in research direction is common aggravates these problems. Busy research group leaders may not properly supervise the people working under them. New graduate students or postdocs may have learned the proper procedures in one research area but now find themselves working in a different area that requires different procedures. In large research teams, different members may develop different systems for recording information. Lack of a common record keeping system and standard operating procedures may cause well-intentioned people to create records that are disorganized, incomplete, and inadequate. Projects that involve teams at different locations, in particular, need common standard operating procedures to help assure that each site obtains comparable results. Often teams do not have clear rules governing interaction with other teams (e.g., who is responsible for taking and storing notes of discussions during teleconferences) leading to poor research coordination and poor records.

Sample management is another area identified by our focus groups that is subject to these types of problems. Complex scientific studies can produce a multitude of intermediary samples requiring additional analysis (e.g., tissue samples from patients, sections of geological drilling cores). Proper labeling (a type of research record) and storage of large numbers of samples is important but can be challenging. Laboratory personnel may not have the training or the resources necessary to perform these functions adequately.

A smaller but significant number of cases involve problems that could be interpreted as attempts to mislead misconduct investigations (missing records and altered or destroyed records). Both RIOs and researchers denounce these cases. Attempts to mislead investigators are considered very serious offenses. In an environment where the records are disorganized and inadequate, it is often difficult to decide if missing records are the result of additional misconduct or simply another example of carelessness.

Discussion

These results clearly show that deficiencies in research records are frequently found during investigations of misconduct cases. Most RIOs at major research universities have been involved with misconduct inquiries or investigations, and these officials found serious problems with the records that hindered their inquiries and investigations. While most of these problems could be resolved with one to three weeks of extra work, many reported problems that were so significant that some issues were never resolved.

The majority of problems center on 2 sets of issues: understandability of the records (e.g., incomplete/inadequate documentation, disorganized records) and access to the records (e.g., physical/electronic access, missing, incomplete, and destroyed records). However, current university policies address access and ownership issues for research data and records rather than the quality of the records (understandability by someone not involved in the project). Our focus group members understood the university’s right to access the data to resolve misconduct issues but chided departments and institutions for not providing more guidance in this other area. A smaller but significant number of cases involve problems that could be interpreted as attempts to mislead misconduct investigations (missing records and altered or destroyed records). The focus group members emphasized that these misleading record keeping practices, whether due to carelessness or malicious intent, were clearly outside the boundaries of good scientific practice and should be avoided.

The focus group members also confirmed that good record keeping is still recognized as an important part of scientific research but cautioned that record keeping in many research projects has become more complex in recent decades as research has moved from a small team working in a single lab to large collaborations involving several research groups at different universities. While they worried about the costs of additional externally imposed requirements, these researchers clearly supported norms of good record keeping and they aided us in the preparation of a ‘Best Practices’ document for academic research record keeping (Schreier et al., 2006).

Recently, research records helped convict Eric Poehlman, an expert on women’s health at the University of Vermont College of Medicine, who admitted to falsifying data on 15 federal grant applications and 17 publications, and has been sentenced to one year in a federal prison for defrauding the federal government (Kinitisch, 2005, Interlandi, 2006). By allowing investigators to act to restore the credibility of published research, this case underscores the importance of good record keeping to the integrity of scientific research. The evidence in the research records clears most researchers of the misconduct allegations. However, disorganized and inadequate records can extend the investigations by several weeks or even months. During this time, on-going research is disrupted. Good research records should allow honest researchers to clear their names and return to their research more quickly.

By documenting some of the types of problems with record keeping that can occur in university-based research, our study provides information that will be useful for policy development at academic institutions. Since our research indicates that problems in recordkeeping can affect the quality and integrity of research, we recommend that institutions, researchers, and research funding organizations proactively take steps to improve research record keeping practices. Institutions should develop policies for record keeping that require good record keeping practices as well as access to data. Institutions and funding agencies should also provide support for record keeping, such as tools, storage areas, and archiving services. Researchers should pay greater attention to good record keeping in their own work, and they should teach good record keeping practices to students, trainees, and personnel.

Our research has some important limitations. First, since we have focused on research records examined during misconduct investigations, our results might not generalize to “normal” record keeping practices. Researchers who are investigated for misconduct may keep poorer records than those who are not. However, since over a quarter of researchers admit to inadequate record keeping (Martinson et al., 2005; DeVries et al., 2006), it seems likely that the kinds of problems we report are not limited to misconduct allegations. Second, we have focused, for the most part, on university officials who are in charge of investigating misconduct. These officials may have encountered a few difficult but very memorable problems while performing their duties. These problems may have unduly affected their perceptions of record-keeping practices. In order to overcome these limitations, we plan to conduct a national survey of university faculty in the next phase of our research on scientific record keeping.

Our current research agenda includes a four-phase project. This article reports results from Phase I (the survey of university research integrity officials) and Phase II (the focus groups with federally funded researchers). While many people involved with research have suspected that there are serious problems with research record keeping, the results reported in this study provide empirical documentation for this belief by showing that poor record keeping is a common source of serious problems in misconduct investigations. Once the types of problems with record keeping in university-based research are empirically established, we can develop a questionnaire that will allow us to determine how widespread these practices are. We are currently developing a national survey of 600 federally funded researchers to assess the level of adherence to good record keeping practices (Phase III) and the degree to which they pass their record keeping practices and standards onto the next scientific generation, their graduate students and post-doctoral fellows (Phase IV). Since there is very little empirical or analytic research on record keeping in science, we also recommend that other researchers and scholars join us in investigating research record keeping in more depth.