Academic Research Record-Keeping: Best Practices for Individuals, Group Leaders, and Institutions

Abstract

During the last half of the 20th century, social and technological changes in academic research groups have challenged traditional research record-keeping practices, making them either insufficient or obsolete. New practices have developed but standards (best practices) are still evolving. Based on the authors’ review and analysis of a number of sources, they present a set of systematically compiled best practices for research record-keeping for academic research groups. These best practices were developed as an adjunct to a research project on research ethics aimed at examining the actual research record-keeping practices of active academic scientists and their impact on research misconduct inquiries.

The best practices differentiate and provide separate standards for three different levels within the university: the individual researcher, the research group leader, and the department/institution. They were developed using a combination of literature reviews, surveys of university integrity officials, focus groups of active researchers, and inspection of university policies on research record-keeping. The authors believe these best practices constitute a “snapshot” of the current normative standards for research records within the academic research community. They are offered as ethical and practical guidelines subject to continuing evolution and not as absolute rules. They may be especially useful in training the next generation of researchers.

Good record-keeping is central to the scientific process.^1–4 Good research records encompass much more that just research data. They include but are not limited to planning and protocol descriptions, data manipulations and analysis procedures, personal and group interpretations of the results, and important communications and group decisions among collaborators. Data management is a subset of the broader concept of research record-keeping. Research records are important for managing and planning research, for replicating results, for documenting collaborations, for publishing and peer review, and for complying with governmental and institutional rules and regulations. In recent decades, legal and regulatory uses of research records have become prominent. Research records have long been used to resolve intellectual property disputes.² However, research misconduct scandals in the 1980s and 1990s involving falsification and fabrication of research records provoked the federal government to require universities to implement research misconduct policies.² Examination of research records is a central feature of inquiries and investigations under such university policies.

We recently conducted a survey of 96 university officials who are responsible for such inquiries and investigations as part of a research project on the impact of research records in research misconduct cases.⁵ We found that over half of the officials who reported experience with misconduct cases at their institutions also reported that they had been hampered in their inquires/investigations by inadequate research records. Also, another recent survey of investigators who had been funded by the National Institutes of Health (NIH) on the prevalence of questionable research practices noted that 27% of the 3,247 respondents admitted to “inadequate record-keeping related to research projects.”⁶

Given the importance of good research records, it is somewhat surprising that formal standards for such records are the exception rather than the rule in academic research laboratories. Although governments have mandated standards for good research records for certain segments of the research community—most notably in the area of human health and safety research through the stringent regulations of the U.S. Food and Drug Administration (FDA) and the Environmental Protection Agency (EPA)^7,8—the majority of academic researchers are not constrained by any external set of record-keeping guidelines. In fact, most academic scientists find the mandated FDA record-keeping practices both onerous and unnecessary. Academic researchers prefer informal guidelines rather than formal standards for record-keeping.

During the last half of the 20th century, technological changes in how records are produced, collected, analyzed and stored, coupled with social changes in the nature of research groups, have created new challenges for research record-keeping.⁹ Traditional practices for such record-keeping are either no longer sufficient or, at worse, obsolete for the modern researcher. New record-keeping practices have arisen to meet these challenges; however, very little research has been done on defining in a systematic way the best practices^* for research record-keeping in the 21st century academy. The literature is replete with descriptions of record-keeping practices, but they are usually personal assessments. In this article, we will discuss the modern challenges to traditional research record-keeping practices. We will then articulate best practices (principles and specific practices) for keeping research records that have emanated from our research and which, we believe, can help researchers respond to these challenges. Because these principles and practices were developed in a systematic way, we believe that they represent current norms within the scientific community (excluding FDA- or EPA-regulated research). We offer these best practices as ethical guidelines for researchers, research group leaders, and research institutions, not as absolute rules or legal requirements.

Modern Challenges to Traditional Record-Keeping Practices

During the 19th and early 20th centuries, a tradition for research record-keeping developed in the academic physical and biological sciences that focused on the research notebook as the primary tool for organizing research projects, planning experiments, recording data, analyzing the results, and storing these records for future reference. Research groups were small, allowing group leaders to personally train young researchers and to regularly examine primary research records; this process ensured the transmission of the record-keeping tradition to the next generation. During the last half of the 20th century, a number of trends in research have complicated the traditional record-keeping process and interfered with the training of new researchers in record-keeping practices.⁹ Chief among these trends were (1) the rise of large research groups; (2) the introduction of research instruments and methods that self-recorded data either on paper printouts or photographically (e.g., X-ray films, slides, and photographs); and (3) the rise of computers in the laboratory.

The current environment for research training, where a graduate student or a postdoctoral fellow is usually a part of the large research group, encourages the teaching of record-keeping practices to new researchers by peers rather than by the senior group leader. Oversight of research is often focused only on the results of a young researcher’s project; it appears to be rare for a researcher’s primary research records to be examined by a senior research group leader. If there are no standards imposed by the research group leader, the actual record-keeping practices within the laboratory group may deviate from accepted norms. The practices actually in use will depend on an individual researcher’s own attitude toward record-keeping and on the attitudes and practices of peers. An extreme example of this occurs when foreign students and postdocs keep their research records in their native languages if there is no requirement to keep records in a common language. There are many anecdotal cases of this practice because international students and postdocs are prevalent in academic research groups. Kanare¹⁰ describes a case where a research group leader discovered, after his Japanese postdoc had returned to Japan, that the postdoc had written his notebooks in Japanese.

The introduction of self-recording research instruments and new photography-based techniques were a major technical advance for science, but they have had the unfortunate byproduct of reducing the utility of the traditional bound research notebook, and more important, of beginning the fragmentation of the research record that is so common today. The first self-recording instruments in the 1940s and 1950s produced paper-based output (they now produce digital files). Photographic recording techniques also appeared at about this time, such as imaging radiolabels on paper chromatographs. These types of records were difficult to incorporate into traditional bound laboratory notebooks due to the cumbersome need to glue or tape such output into notebooks and the sheer volume of such records. Many research groups abandoned the traditional bound notebook in favor of loose-leaf binders or manila folders for both the collection of data output (instrument and photographic data) and for the planning of experiments. Soon, the processing and analysis of data was also placed into loose-leaf binders or manila folders, thus contributing to the physical fragmentation of research records.

The rise of computers in the laboratory was another major technical advance. They made possible the collection and processing of monumental amounts of data that were critical for the advancement of many fields (e.g., X-ray crystallography, genomics, and proteomics). However, the computer also promoted the fragmentation of the research records in academic laboratories and made oversight more difficult. Researchers routinely use many different computer applications in the course of their work and produce a variety of research records (e.g., word processing files, spreadsheets, instrument data output files, etc.). For a single research project, relevant computer files can be scattered among several computers within a research group, depending on the number of individuals working on the project. Within each computer, the pertinent files may be scattered among apparently unrelated folders on the hard drive. In addition, the rapid obsolescence of computer software and hardware (often only three to five years per generation) has made the long-term electronic retrieval of research records problematic, if not impossible.

Technological Solutions to Modern Record-Keeping Challenges: Different Responses of Academia and Industry

Computer technology has also been called upon to provide solutions to these modern record-keeping challenges. Laboratory information management systems (LIMS) and electronic laboratory notebooks (ELNs) are the two major types of tools that have been developed.^11–13 Industry has embraced these systems due, in part, to intellectual property concerns and stringent FDA regulations. LIMS systems in particular have been extremely helpful in managing laboratories that generate a great deal of data and use standardized procedures, which is often the case in industrial and clinical labs and pharmaceutical quality control facilities. However, in academic (discovery-type) research, protocols and methods can change frequently, and the amount of data collected tends to be relatively limited. Because LIMS systems need to be customized for each process they handle, such systems are therefore viewed as being too inflexible for the typical academic laboratory.¹² The exceptions here are areas like genomics and proteomics, whose procedures have become highly standardized with a need to process high amounts of data. ELNs supposedly mimic the bound paper notebook, but again due to flexibility issues, anecdotal evidence suggests that such tools are not commonly used in academic laboratories.¹²

These technological solutions themselves are limited by the basic assumptions used in their design about what constitutes good records and good record-keeping. The best practice principles that we developed and have described below can be used to guide the development of future generations of LIMS and ELNs. We believe that these principles transcend the mechanism for their implementation, whether they be paper-or computer-based.

Genesis of the Best Practices

We developed the best practices described below as an adjunct to our research-on-research-ethics project entitled “Scientific Record Keeping and Responsible Research Conduct” funded by a grant from the National Science Foundation (NSF).⁵ We carried out the research to determine the actual record-keeping practices of active NSF- and NIH-funded scientists and how these influence the adjudication of research misconduct investigations. To help prepare a national survey of active NSF and NIH scientists on their record-keeping practices and to aid in the analysis of the responses, it was necessary to have a set of generally recognized best practices, that is, normative standards for research record-keeping. We derived these best practices from four sources: (1) the research training and ethics literature; (2) institutional policies and guidelines; (3) interviews with university officials responsible for investigating allegations of scientific misconduct; and (4) focus groups composed of faculty members from both medical and nonmedical faculties of Duke University, East Carolina University, North Carolina State University, and the University of North Carolina at Chapel Hill.

The research training and ethics literature was our primary source of record-keeping practices. The regular research literature (e.g., journal articles, monographs, conference proceedings, etc.) was not excluded from our review; however, this literature usually does not discuss research record-keeping practices. The ethics and research training literature is large and often included detailed and discipline specific recommendations for record-keeping. Many of the references for record-keeping practices were in textbooks or primers for students learning laboratory techniques for the first time. They usually focused on the proper construction of a research notebook and more, recently, its electronic counterparts. The monograph Writing the Laboratory Notebook by Kanare¹⁰ is the acknowledged classic in this field. The practices described in this book and the other references are the practices that an individual researcher should follow when he or she is planning, collecting, recording, and analyzing research. We named these specific practices individual best practices. Only recently have references appeared that speak to higher-level best practices, such as those that research group leaders, departments and institutions could emulate to promote good record-keeping. Selected primary documents used in our compilation of our best practices are listed in the references at the end of this article.^{2–4,9,10,14–25}

Institutional policies and guidelines that deal with research records from many research universities were reviewed. Although our review was far from inclusive, we found that such policies and guidelines usually spoke to issues of retention, ownership, access and transfer of research records. Only rarely did a policy or guideline (e.g., those of the University of California, San Francisco)¹⁸ set specific standards for record-keeping beyond these administrative issues. Sources that contained representative policies and guidelines that were useful are noted in the reference list.^18–25

Our research ethics project included interviews with university officials from 96 research universities.⁵ These officials provided us with a number of positive practices for good record-keeping based on their experiences. In almost every case, however, these practices reiterate those that appeared in the literature. Thus, the university officials provided anecdotal evidence for the value of these practices.

As the number of specific best practices grew, they became too unwieldy to be a useful guide if compiled into a single list. An organizational structure was needed. We divided the best practices into three hierarchical groups: best practices for individual researchers, best practices for research group leaders, and best practices for departments and institutions. We also developed summaries—essentially statements of principles for the best practices for each group (see Lists 1, 2, and 3)—and compiled the detailed best practices separately. The resulting document, containing both principles and specific practices, was provided to the focus groups for their review and suggested revisions.

List 1.

Best Practice Principles for Individual Researchers^*

Useful (good) research records explain what you did, when you did it, why you did it, how you did it, who you are (the person creating the record), what project(s) it was a part of, who thought of it if not you, what special materials and instruments you used, where you obtained the materials and instruments, what happened and what did not happen (data), how you manipulated and analyzed the results, your interpretation (and the interpretations of others if important), and what will be the next steps in the project based on these results.

In addition, good research records are legible if handwritten, are recorded using reliable materials and tools, are well organized (e.g., well labeled, indexed, catalogued, etc.), are accurate and complete; they include (1) all original data and important study details (meta-data) and (2) successful and unsuccessful studies and activities, describe and date all alterations and changes in records, allow repetition of your procedures and studies by yourself and others, are accessible (physically and/or electronically) to others both short term and long term, are stored and backed-up properly for the short and long term (archiving), are witnessed where needed to protect intellectual property rights, are in compliance with departmental, institutional, and federal regulatory requirements, with special care given to human and animal research, and are the research diaries of the researcher’s work and thoughts.

^*Researchers may be at any level from student and staff to senior faculty if they are personally performing hands-on research. Research records are defined briefly as recorded information, regardless of media, that is necessary for the reconstruction and evaluation of the research. An individual record element may not need all the above attributes, but the whole record probably does.

Source: This list adopted from Table 11.1 from Scientific Integrity: An Introduction with Case Studies, by Francis L. Macrina, ASM Press (2000), with permission.

List 2.

Best Practice Principles for Leaders of Research Groups

Research group leaders should set standards for record-keeping practices for individuals in their group in areas such as research studies/activities within the group (handwritten and electronic notes, data, and other documentation), labeling and cataloging of experimental samples, tangible products of research, etc., communications with collaborating researchers, such as letters, e-mails, minutes of meetings (face-to-face or teleconference), etc., provide/assure that group members receive training in record-keeping practices, provide motivation by emphasizing the benefits of good records and the problems associated with poor records, provide examples of well-maintained records and good record-keeping practices, clarify data and research record ownership and access rights, perform periodic reviews of the records of the members of your group, delegate, as needed, oversight and training duties for group records to senior members of your group and perform periodic checks on the performance of these duties and modify/reassign duties as needed, provide the tools (paper-based notebooks or electronic hardware/software), establish temporary storage areas for records in use (both paper and electronic) and appropriate backup facilities/methods, require adherence to group record-keeping standards by group members, promote communication of research information within the group, have a plan to assure the transmission of important research information (accessible and understandable records/notebooks) from departing group members, require adherence to departmental, institutional, and legal requirements, seek to assure the long-term accessibility of records for a set period of time (archiving) after completion of the research, and update records standards as needed.

List 3.

Departmental and Institutional Best Practices for Research Record-keeping

Department/School Level Make available training/mentoring to faculty (especially new faculty) on research group/lab management skills and practices. Include best practices in scientific record-keeping. Encourage faculty members to have a strategy/plan (preferably written) for research record-keeping. The strategy should adopt “individual best practices” to the research group’s circumstances and include active mentoring and oversight of trainees and staff. Provide record-keeping materials and resources if possible (e.g., research journals, lab notebooks, specialized software, bar-coding equipment, dedicated servers [computers] for storing electronic records.

University Level Provide clear policies on research record ownership, access, retention, transfer, and destruction. What constitutes research records should be clearly defined and should include paper, electronic, and tangible forms of research information. Provide Institutional facilities for archiving records (all media). Many institutions already have this service through their university libraries. To help assert university ownership of research records, provide record-keeping materials for the departments/units to distribute (e.g., research journals, lab notebooks, specialized software, dedicated servers [computers] for storing electronic records). Provide resources to help departments/units provide training in research record-keeping (e.g., training materials, examples of good records and practices, a Web-based tutorial in record-keeping practices). Provide resources to help assure the long-term accessibility of electronic records (i.e., help protect against hardware/software obsolescence and availability issues for older data and records).

The focus groups were particularly helpful in refining the best practices. The focus-group members totaled 48 active senior scientists (12 on each campus), who have NSF or NIH funding in a variety of research fields (from basic sciences like physics and chemistry through clinical fields to social and behavioral sciences). Focus-group sessions of four hours in length were held in the spring and summer of 2004. Interestingly, the members of the groups directed most of their comments toward best practices for research group leaders and departments and institutions, the areas that were least discussed in the literature. They stressed the pivotal role of the research group leader in setting standards and procedures for his or her group. The focus-group members identified some new types of research records that had previously not been mentioned in the literature, such as records of e-mails and teleconference transcripts and minutes. These types of records have greatly increased in importance due to the highly collaborative nature of modern research, where the collaborating scientists and their research groups can be physically quite distant from each other. Records of communications about research results, plans for future research, and decisions regarding results or future plans need to be documented to prevent misunderstandings and miscommunications. Another area of extreme importance to all focus groups was the concern that any set of best practices resulting from our work would be adopted as mandated requirements by federal agencies, resulting in inflexible and onerous regulation of their research.

The Best Practices

Description

Lists 1, 2, and 3 summarize the best practice principles for research record-keeping at all three levels described above. A complete document with detailed best practices is available at 〈http://www.research2.ecu.edu/BestPractices_Sep04.doc〉.

Reflections

The best practices described above and in the three lists are a synthesis of traditional and modern practices from a variety of sources. They emphasize principles over detailed practices, which are subject to obsolescence and disciplinary restrictions. They also emphasize the critical role of the research group leader in setting standards and in the training of new researchers. In addition, they explicitly acknowledge the role of departments and institutions in setting the organizational context within which the research groups operate. Our focus-group members felt that these two higher-level sets of practice principles were applicable to any research discipline or institution. These principles address the effective management of scientific laboratories and work groups, and are aimed at promoting good communication between current research participants and future participants (through stored records).

It is our opinion that a breakdown in the training and supervision of new researchers has contributed to the decline of research record-keeping. The fragmented and still-evolving nature of modern research records has contributed to this breakdown by making many of the old record-keeping practices obsolete, with no generally accepted replacements in sight. Young scientists in training and new research group leaders may find it difficult to understand the linkages between the specific, detailed processes they follow in their research and the general norms of scientific research. By identifying a separate set of responsibilities assigned to the research group leader, they may be able to more clearly understand how to manage their groups and train their students more effectively.

We present this new synthesis of best practices to the research community as a potential aid in thinking about the challenges of record-keeping in the first decades of the 21st century. We hope that it will guide the development of better practices among research group leaders, as well as encourage departments and institutions to adopt practices and policies that will aid research group leaders in their responsibility to keep good records of their research.

Biography

Dr. Schreier is a university administrator charged with coordinating university research compliance activities including the review of allegations of research misconduct. He is primarily responsible for the development of the best practices document described in the present article. Dr. Wilson, the principal investigator of the NSF grant, is an experienced survey sociologist who is primarily responsible for the development and execution of the surveys and focus groups employed in the NSF project. Dr. Resnik, a bioethicist formerly on the faculty of East Carolina University, provided the initial ideas for the NSF project as well as critical review of all steps in the project.