Preparing for Clinical Trial Data Audits

In the mid-1950s, the National Cancer Institute (NCI; Bethesda, Maryland) began funding the Cooperative Group Program. Throughout the next 25 years, there was no on-site verification of protocol compliance and validation of the data submitted by participating members of these cooperative groups. In contrast, the US Food and Drug Administration (FDA; Washington, DC) required that pharmaceutical companies perform on-site validation of the clinical trial data involving testing of investigational new drugs. In 1979, it was discovered that scientific fraud had occurred at one institution¹ that was a prominent member of a major NCI-funded cooperative group. As a result of that event, in 1981 the NCI implemented a requirement that all entities funded by the NCI must have a system in place for on-site auditing of clinical trial data and protocol compliance. Verification of administrative requirements, such as oversight by an institutional review board (IRB) and documentation of written informed consent, was also implemented as part of this auditing process.

Over the subsequent 25 years, the process has been refined and supplemented with additional requirements such as the auditing of consent form contents and the handling of investigational drugs in institutional pharmacies. Audits by various entities (the NCI, the FDA, pharmaceutical firms, etc) are a fact of life,² and fortunately, the process has rarely uncovered instances of scientific fraud. The more important benefit of auditing is that the scrutiny applied by external review of those involved in clinical investigation often results in better compliance with trial requirements and greater accuracy in data collection, to the benefit of all clinical research. Audits also serve as a hands-on educational process for both physicians as well as clinical research associates (CRAs), both when being audited and when invited to be the auditor at another site in their cooperative group.

The announcement that an audit will be conducted in the near future always creates dread and anxiety in those persons to be visited, especially in those closest to the data collection process—the CRAs. In this article, we provide some hints for audit preparation and ways to avoid deficiencies. We draw on our experience with the Cancer and Leukemia Group B (CALGB) audit process for our frequently asked questions. Because all NCI-funded cooperative groups have similar audit programs, the information can be generalized for any such audits, though there may be minor variations depending on the cooperative group involved.

Question: What areas will be audited?

A document that should be part of the training of all new CRAs and all clinical investigators is the Audit Guidelines of the NCI Clinical Trials Monitoring Branch.³ This document provides the standards that all cooperative group auditors must follow. When an audit is scheduled, a review of this document will help the site staff understand what items will be reviewed and how deficiencies will be assessed and graded. There are three main areas of review: (1) IRB oversight and consent contents; (2) handling of investigational drugs, including a visit to the pharmacy; and (3) patient case review. In the latter, there are six subcategories of review: (1) consent form signing, dating, and filling in blanks; (2) protocol eligibility; (3) protocol-directed treatment, including (when applicable) compliance with radiation therapy and surgery technique; (4) verification of treatment response (or lack thereof) and patient outcome; (5) toxicity grading and recording; and (6) accuracy of data recording and submission, and the fulfillment of special requirements (when applicable) such as submission of blood or urine samples, quality-of-life questionnaires, pathology specimens, etc.

Question: Which patients and studies will be audited?

This varies with each cooperative group, but the NCI requires review of at least several studies involving investigational drugs, and studies representing a cross-section of protocols on which the site has enrolled patients. In the CALGB, an effort is also made to include at least one patient entered by each participating physician at the site, and to include some studies involving complex treatments (such as for acute leukemia) and combined-modality treatment.

Question: Is there any source of information regarding audit preparation?

Each cooperative group maintains a set of policies regarding how audits are conducted, all in accordance with the NCI Audit Guidelines. In the CALGB, there is discussion of audit preparation at CRA workshops, and once yearly at the main group meeting, an “Audit Preparation Workshop” is held. The slide presentations for this workshop are also available to group members on the CALGB Web site. Other groups have similar training sessions for both group members to be audited and those who will be conducting an audit.

Question: What physical facilities and staff should be arranged?

Depending on the size of the audit team, a room should be reserved that is large enough for the team and the local staff. At a few CALGB audits, only a desk and chair in an office have been set aside for a team of six people, which is obviously insufficient. A CRA experienced with both the case report forms and the local medical records should be immediately available to assist in finding items to be reviewed and answering questions the auditors may have. Some cooperative groups do not want any local staff on hand during an audit, but at CALGB audits, a local staff person is required to stay with the audit team all day. A senior physician should also make time available for the exit interview at the end of the audit.

Question: What will be audited regarding the IRB?

All studies on the Audit Patient List will have an assessment of compliance in submitting the protocol to the IRB for initial review, annual renewals (by each 365-day interval since the previous approval), review of all relevant amendments/revisions/updates within the NCI-required 90 days after implementation, and timely submission and review of all reportable local serious adverse events and those broadcast by the relevant group or other research entity (e.g., NCI, pharmaceutical firms). The CALGB auditors have found that contemporaneous updating of such IRB documents with chronological filing in specific loose-leaf binders allows one to be certain that all such items are easy to locate (for both local staff and auditors) and that nothing is missing.

Not only will the studies listed on the Audit Patient List be reviewed for compliance with IRB requirements, but in some groups (including CALGB), several unannounced closed studies are also audited for continued fulfillment of IRB review requirements.

Question: What will be audited regarding consent form contents?

Review of the details of a sample of the local consent forms is required by the NCI. A minimum of three forms must be reviewed, but most sites have a sufficiently large patient accrual so that twice this number are reviewed. Some groups require that the signed consent form be submitted to the group headquarters for review during patient registration. At that time, any deficiencies are identified for correction. Other groups depend on the audit process to monitor compliance with consent form contents. There are many requirements for consent form content specified in Federal government documents, but the NCI requires two items to be identical to the protocol-specific Model Consent Forms (approved by the NCI staff and/or the central IRB before a new study can be activated); they are: (1) all the risks listed in the Model Consent Form must be in the local consent form; none can be omitted; (2) all the alternatives and associated explanations in the Model Consent Form must be present. It is worthwhile to check all local consent forms to be certain such items are not omitted.

One error that occurs commonly is the failure to complete fill-in-the-blank items on a consent form. If the local consent form has additional items such as patient initials on every page, names and phone numbers, and yes/no responses for special items such as blood sample collection, then they must be completed. It is a simple task to be certain that these items are completed at the time a patient signs the original consent form. No patient should be registered until local staff members have verified that all such items have been completed.

Question: What will be reviewed at the pharmacy or other drug preparation site?

Since 1983, the NCI has required that cooperative group audits include a review of the handling of investigational drugs. The Audit Guidelines³ and the NCI Web site⁴ have information regarding the requirements for such handling, as well as the “Dos and Don'ts.” Others⁵ have also published guides on preparation for this aspect of audits. The pharmacist should assemble all Drug Accountability Record Forms (DARFs) relevant to the audit and all invoices for all incoming and return shipments, and arrange them in chronological order for auditor review. Someone familiar with the investigational drugs must be available on the day of the audit and plan on meeting with the audit team. In-stock investigational drug supplies should be checked to ensure that the containers are properly labeled as to drug and study, and that the amount of remaining stock corresponds to the figures on the DARF.

The CALGB Pharmacy Committee has prepared a checklist (Table 1)that can be a useful guide for pharmacists in making sure deficiencies are minimized prior to the actual audit. One error seen repeatedly in audits is that excess drug supplies are kept well after they are needed (i.e., the study has closed to new patient accrual and no patients are being treated, or the drug has expired), and only a few days prior to the audit, they are returned to the NCI. The pharmacy staff should have a system in place whereby shelf stocks are assessed on a monthly or quarterly basis, at which time the return of unused drug supplies is accomplished.

Table 1.

Are You Ready for a Pharmacy Audit?

Are your NCI DARFs completely and correctly filled out? •There is a DARF for each investigational drug and dose strength. •All pertinent information has been completed on the DARF (i.e., no blanks). •No correction fluid (“white-out”) has been used, and all corrections are lined out and initialed. •The drug supply balance listed on the DARF matches the shelf stock. All DARFs are arranged in chronological order and by drug.

Are pharmacy satellite records available and properly maintained? •A DARF is completed at the satellite facility for any drug supplies stored for multiple therapy doses, and the incoming and outgoing supplies must match the main member DARFs.

Are all shipping invoices and transfer approvals available and in chronological order? •INDs cannot be transferred from one supply to another without NCI (or drug company, when relevant) approval. Documentation of transfer approval must be retained. •The invoices for all incoming and outgoing shipments must be available and in chronological order.

Have excess drug supplies been properly returned or destroyed? •Expired drug should be returned to the supplier in a timely interval. •When a study closes, excess IND supplies for the study should be transferred to another open study, if available, with NCI approval or returned to the supplier in a timely interval. •If a patient returns excess dispensed drug, it should be destroyed in accordance with NCI or drug company requirements. The drug amounts should not be re-entered on the DARF, as the DARF should denote only drug that can be dispensed to a patient.

Are INDs stored appropriately and securely? •INDs are stored separately by protocol and strength, with containers labeled properly. •INDs are stored separately from commercial supplies. •INDs are stored in an area accessible only to authorized personnel, or are kept in locked cabinets/refrigerators during times when the storage room itself can be accessed by unauthorized persons.

Abbreviations: NCI, National Cancer Institute; DARF, Drug Accountability Record Form; IND, investigational new drug.

Question: How should the local records be prepared for the patient case review?

The work of the auditors is greatly facilitated if the important items regarding protocol compliance and patient management have color-coded adhesive tabs applied for ease of identification. For example, one color would indicate pre-enrollment and eligibility items, another color would indicate each treatment cycle and toxicity assessment, and still another color to represent the salient radiographic reports regarding response, etc. Importantly, each tab must be identified. In some audits, the local staff members have appended such color-coded tabs, but nothing was identified. The local records thus have multiple colored tabs, but without identification, it is still difficult for the auditors to find particular items.

Some institutions now have electronic medical records, and nothing on paper is available for review. In such cases, it is worthwhile to print the most important items for review (e.g., pathology reports, initial physician evaluations, operative reports). The site staff should then have a computer screen available for each auditor or team, with a local staff member sitting adjacent who is qualified to locate items in the system as they are needed. The auditors could then, for example, request the baseline set of blood test results, and they would be immediately brought up on the screen. The same process can be used to review electronic records of radiographs. Computed tomography scans are now recorded on disc and not on film in most hospitals. In the CALGB, the auditors rarely review the normal radiographic studies (e.g., a screening brain magnetic resonance imaging study), but the auditors do want to review the serial scans done on a patient whose assessable tumor is being evaluated for treatment response. Thus, relevant films (or discs) should be made available in such instances. Again, if electronic records are to be reviewed, a qualified local person must be available to assist with the process.

When staff are preparing for an audit, asking questions of the cooperative group staff responsible for auditing is always encouraged. Clarification of the audit process and requirements can also be provided by NCI staff at the Clinical Trials Monitoring Branch (telephone: 301-496-0510).