Quality assessment of phase I dose-finding cancer trials: proposal of a checklist

Abstract

Background

Qualitative checklists for phase III trials have been proposed, to improve the reporting of such trials and to assess the validity of their results.

Purpose

Our objective was to develop such a scale for phase I cancer trials.

Methods

From a review of existing guidelines and checklists for phase III clinical trials, a staff team was responsible for the first selection of items and the construction of the questionnaire. The proposed quality assessment measures were rated by the survey respondents comprised of phase I research clinicians and statisticians on a 4-point Likert scale. Selected items from the quantitative analysis of the questionnaires were reviewed by an expert team who was responsible for providing the final items list. This was then applied to 103 recently published cancer phase I trials.

Results

Of the 48 initial items proposed by the staff team, 17 were selected from the quantitative analysis of the 99 participants’ ratings. After qualitative analysis by the expert team, a 15-item checklist was derived, with 5 items related to trial objective, 5 to design, and 5 to analysis. The application to 103 recent journal articles on phase I cancer trials evaluating cytotoxic drugs showed on average the report of 10 items (range: 6–13) with 4 items reported in more than 95% of papers, while 2 were poorly reported.

Limitations

The response rate of participants was 20.7%.

Conclusions

A quality assessment checklist was developed for improved critical appraisal of the reporting of cytotoxic, dose-finding phase I oncology trials. This may be a first step toward a minimum standard of quality measures for all phase I clinical trial reports.

Introduction

Dose–finding phase I trials are first-in-man studies, designed to determine maximum tolerated dose (MTD) and to assess safety profile. It is recognized that cytotoxicity directed towards the cancer cell is typically associated with undesired toxicity on some normal cells and tissues. The need to achieve a high kill potential of cytotoxic drugs on the malignant cells is counterbalanced with the need to minimize toxicity to a tolerable and acceptable level. Thus, the aim of the phase I trials is typically to determine, as a surrogate of effective dose, a maximally tolerable dose with an acceptable toxicity profile which can then inform the design of subsequent trials and be further investigated in expanded phase II investigations. This is referred as the MTD or the recommended phase II trial dose (RPTD) [1]. Typically, the MTD is the dose level immediately below the dose level at which toxicity pre-determined to be dose-limiting occurs among a pre-defined number of trial subjects. Traditional phase I designs are thus based on the assumption that both efficacy and toxicity increase with dose. The goal is to both minimize the number of subjects exposed at low doses that may result in under-dosing and an ineffectual anti-tumor effect and to minimize the chance that subjects will be exposed to excessively toxic or lethal doses. Therefore, most phase I designs are adaptive dose-escalation schemes, in which the dose is gradually increased throughout the experiment until the MTD (or the RPTD) is supposed to be found. It is usually defined as the dose level associated with a predetermined proportion of patients experiencing dose limiting toxicity (DLT).

For the past 20 years, phase I oncology trials have been the subject of considerable ethical [2,3] and statistical debate [4,5]. The number of available designs has much grown, and beside the mostly used algorithm-based designs such as the standard ‘3+3’, innovative model-based designs have been proposed [6–8]. These latter designs define the MTD as a dose with a certain probability of DLT. Whatever the design, the overall toxic death rate due to toxicity for subjects in cancer phase I studies has been reported as low as 0.49% [9], decreasing from 1.1% over 1991–1994 to 0.06% over 1999–2002 [10], suggesting that ethical concerns have been addressed. To learn from this experience in the phase I oncology trial literature and to make improvements upon future trial designs, it is important to be able assess the quality of methodology of these trials.

We performed a Medline search using key words ‘quality’, ‘checklist’, and ‘phase I’ of English-language journals of work on quality assessments of Phase I trials. Besides articles related to the establishment of Good Manufacturing Practice or Good Laboratory Practice in regulatory settings, only two papers were found. One article authored by Chang et al. provided recommended guidelines to standardize the reporting of phase I and phase II neuro-oncology trials [11] in regards to title, introduction, methods, results, and discussion sections of such reports. The other article evaluated the presentation of abstracts at the annual American Society of Clinical Oncology (ASCO) meeting [12] focusing primarily on the subsequent publication of these abstracts, concluding the under-reporting of final results of phase I oncology trials.

This dearth of published guidelines for assessing the reporting of phase I oncology trials contrasts with the phase III trial setting where many quality assessment lists for trial reporting have been developed [13,14]. However, these scales of assessment cannot apply directly to phase I reporting. One reason these Phase III assessment measures may not be applicable to assessing Phase I trials is greater similarity across trial designs for Phase III trials that are often comprised of two parallel treatment arms, whereas Phase I trials employ serial dose-escalation cohorts and increasing dosing levels. Therefore, quality assessment scales for phase I cancer trials should be specifically created. Nevertheless, ways to evaluate toxicity experienced by subjects in the phase III trials [15] could be applied to phase I trial assessment.

The main aim of this article was to provide a qualitative checklist for phase I oncology studies, to improve the reporting of such trials, enabling readers to understand and to critically appraise such trial’s design, conduct, analysis, and interpretation, as well as to assess the validity of their results. We were most interested in determining how quality can be best measured. Thus, we chose consensus decision-making techniques in creating this Phase I trial assessment checklist utilizing the knowledge and experience of experts in the fields of oncology, statistics, clinical research, as well as the limited available evidence in the literature [14]. We further applied the developed checklist to 103 recently published papers reporting phase I oncology clinical trials that evaluated cytotoxic investigational drugs.

Materials and methods

Methods

Staff Team

The staff team was comprised of all authors (with the exception of KC and AI), who are all involved in the conduct and analysis of phase I trials, including phase I oncology trials. The staff team members, two of whom are clinicians (QL and VL, in which one was an hematologist – VL) and two of whom are statisticians (SC and SZ), were responsible: (1) for the first selection of quality assessment measures to be considered as items on the phase I trial reporting checklist; (2) for the creation of the questionnaire of these items that was then presented to the survey participants, as described further; and (3) for the analysis of these survey results.

Selection of measures of assessment of phase I reporting

First, we collected all items from existing guidelines or checklists developed for RCTs – including CONSORT (Consolidated Standards of Reporting Trials) [16], ignoring those related to randomization, which are not relevant for phase I trial assessments. Then, new questions were generated to address the specific issues related to phase I trials, resulting in a total number of 48 items in the questionnaire (Appendix 1).

Selection of survey participants

The participants were required to be clinicians or statisticians involved in phase I oncology clinical trials, as previously reported [13,14,17–21]. First, we selected all first (or co-) authors of an original published paper reporting phase I trial or phase I methodology from January 1999 to April 2005. This was based on a Medline search, using the keywords ‘Clinical Trials, Phase I’ [MeSH] (Medical Subject Headings) and ‘cancer’ [22]. All potential participants were contacted individually by e-mail, with a follow-up reminder about two weeks later as needed. Each participant was instructed to fill out a web-based questionnaire and to rank for inclusion of these quality assessment measures into the criteria listing for phase I oncology trials, using a 4-point Likert scale, in which the four potential choices for each item was scored as follows: strongly agree that the item should be included (2 points); moderately agree (1 point); neutral (0 point); disagree (−1 point). To ensure a complete data set per participant, each survey participant was required to provide a ranked response to each and every item of the 48 items in the survey.

Analysis

The analysis of the questionnaires was both quantitative and qualitative. To reduce the number of items, we used quantitative analyses per the Delphi consensus technique, as previously done for quality assessment checklists for phase III trials [14].

The selection of each item to be included in the final listing of phase I quality assessment items was based on three criteria. The first criterion was the percentage of survey participants who strongly agreed with the inclusion of each specific item in the phase I checklist. The second criterion was the percentage of survey participants who ranked an item with either strong or moderate agreement for inclusion. The third criterion was the mean score for each item surveyed on a 4-point Likert scale (with N=99 for each distribution). For each of these three criteria (percentage strongly agreeing for checklist inclusion, percentage agreeing (strongly or moderately) for inclusion, and mean score for each item) the frequency distribution across the 48 items was derived, and the third quartile (Q3) was designated as the cut-off point for inclusion in the final checklist. Only quality assessment items that had at least one of the three criteria above the Q3 value were selected. This was performed to reduce the number of items presented to the expert team (KC and AI) for the final review.

The expert team was responsible for providing the final items list through a qualitative analysis of the items previously selected. Specifically, they were asked whether they preferred the rewording or the original phrasing of each selected item. They were also given the opportunity to add extra items that received a second chance to be included into the final criteria list.

Assessment of the quality of reporting of phase I cancer trials in peer-reviewed journal articles

We reviewed all the consecutive articles from the list of journals shown in Table 3, based on a Medline search with publication date from January 1, to July 1, 2005, and identified all articles describing phase I cytotoxic, dose-finding cancer trials. We only considered peer-review journals that published more than one phase I article on the time interval.

Table 3.

Characteristics of the 103 journal articles on phase I oncology trials which were evaluated for meeting quality assessment measures in the newly designed phase I trial checklist

	N(%)
Journal
Clin Cancer Res	18 (17.5%)
Cancer Chemother Pharmacol	14 (13.6%)
J Clin Oncol	13 (12.6%)
Br J Cancer	8 (7.8%)
Ann Oncol	6 (5.8%)
Invest New Drugs	6 (5.8%)
Oncology	6 (5.8%)
Cancer	5 (4.9%)
Anticancer Res	4 (3.9%)
Cancer Invest	4 (3.9%)
Chemotherapy	4 (3.9%)
Eur J Cancer	4 (3.9%)
Anticancer Drugs	3 (2.9%)
Int J Radiat Oncol Biol Phys	3 (2.9%)
Jpn J Clin Oncol	3 (2.9%)
Gynecol Oncol	2 (1.9%)
Geographic region
Asia and Oceanic ocean	18 (17.5%)
Europe	41 (39.8%)
North America	44 (42.7%)
Trial
Sample size: median (Q1–Q3)	25 (18.5–33.5)
Number of dose levels: median (Q1–Q3)	5 (4–6)
Treatment type
Monotherapy	40 (38.8%)
Combination	63 (61.2%)

Statistical considerations

Summary statistics (median with range) were used to describe journal article characteristics, characteristics of the survey participants, and the data extraction results. Credit sum, defined as the number of reported items from the final list, were computed on the whole set of questionnaires, then stratified according to the status of the survey participant, either physician or statistician. Comparison of credit sums across subgroups was based on the nonparametric Wilcoxon rank sum test or Kruskal–Wallis rank sum test when necessary.

Statistical analysis was performed on R software [23].

Results

Survey participants

We identified and contacted via email 809 authors of articles that reported on phase I oncology trials and were published in English-language, peer-reviewed journals between January 1999 and April 2005. Of the 809 attempted email contacts, 332 (41%) never generated a reply, 296 (36.6%) of the emails were returned due to incorrect address, 26 (3.2%) of the potential participating authors declined on the basis of considering themselves to lack the necessary expertise in the field, while 10 (1.2%) declined citing lack of time to participate. From the 477 remaining potential participants a total of 99 (20.7%) authors fully responded to and completed the questionnaire on the website (see Appendix 1 for this questionnaire). Of these 99 survey respondents, 73 were physicians and 22 were statisticians (See Table 1 for main characteristics of the 99 survey participants).

Table 1.

Main characteristics of the 99 participants

	N(%)
Asia and Oceanic ocean
Australia	1 (1.0%)
China	1 (1.0%)
Israel	1 (1.0%)
Japan	8 (8.1%)
Singapore	1 (1.0%)
Europe
Belgium	1(1.0%)
Denmark	2 (2.0%)
France	16 (16.2%)
Italy	5 (5.1%)
Germany	5 (5.1%)
Greece	3 (3.0%)
Spain	1 (1.0%)
Sweden	1 (1.0%)
Switzerland	5 (5.1%)
UK	4 (4.0%)
North America
Canada	2 (2.0%)
USA	42 (42.4%)
Status
Statistician	22 (22.2%)
Physician	73 (73.8%)
Other	4 (4.0%)
Employer
Academic	80 (80.8%)
Industry	9 (9.1%)
Other	10 (10.1%)
Experience in the conduct of phase I trials
<5 years	9 (9.1%)
5–10 years	34 (34.4%)
10–15 years	23 (23.2%)
>15 years	32 (32.3%)
Not specified	1 (1.0%)

The mean rating score on the Likert scale was above 1.01 for one half of the surveyed quality assessment measures, while one fourth of these items had a mean rating score value above 1.40. Based upon these results, 10 items from the survey were selected as fulfilling the cutoff threshold value for the three designated criteria. These factor criteria were: percentage of strong agreement for item inclusion above 54% of surveyed participants; percentage of agreement above 91%; and mean score above 1.40. Three additional survey measures fulfilled two of these criteria, and per the predetermined selection guidelines, they were also chosen for the final survey review. Lastly, two remaining survey items – one related to the justification of the starting dose and one related to the report of drug-related deaths – met one of the criteria and these were also added to the final draft checklist.

This resulted in 15 selected items. Stratified analysis of the survey results of the subset of 22 statisticians allowed selection of two additional items: how administered doses matched the dose-allocation method for the trial and the possibility of withdrawals in the results. Subset analysis restricted to the 73 physicians’ responses did not select out any further item for inclusion in the final draft checklist.

The staff team then submitted the resultant 17 item questionnaire to the expert team. After rewriting and discussion, 15 questions – five related to trial objectives, five to clinical trial design, and five to analysis were retained for the final qualitative assessment checklist for phase I cytotoxic, dose-finding, oncology trials (Table 2).

Table 2.

Final 15 question quality checklist resulting from the whole selection procedure, with percentage of actual reporting in the 103 peer-reviewed journal articles describing phase I oncology trials published from January 1999 to April 2005

Focus	Questionnaire for reporting phase I dose-finding cytotoxic oncology clinical trials	Percentage of reporting
Objective	Was the objective to find a MTD or a recommended dose for phase II trials?	95.1
	Was the MTD associated with a prespecified toxicity rate?	80.6
	Was the toxicity related to the drug only used to define the MTD?	70.9
	Was the measure of toxicity based on international grading systems such as OMS or NCI scales?	95.1
	Was the trial disease oriented?	53.4
Design	Was the starting dose specified?	99.0
	Was the first dose level justified from preclinical or clinical data?	28.2
	Was the number of dose levels clearly mentioned in the study design?	97.1
	Did the choice of the distinct levels explained?	48.5
	Was the dose allocation method clearly described?	89.3
Analysis	Were all included patients analyzed?	72.8
	Were doses and responses clearly reported?	68.9
	Did the dose allocation along the trial match the described method?	45.6
	Was the estimated MTD or the recommended dose level associated with a measure of variability?1	0.0
	Was the process of estimating the MTD or recommending the dose for future trials clearly explained?	56.3

¹One expert suggested dropping this item.

Evaluation of new checklist on published articles

Our Medline search of journal articles published from January 1, 2005 to July 1, 2005 led to the identification of 154 published articles on phase I cancer trials eligible for further review, which eliminated 51 of these articles from the sample, based on our criterion that the sampled trial reports involve testing of cytotoxic drugs (41 involved noncytotoxic agents) and be specifically phase I (10 were not). This left 103 remaining articles found in 16 journals (Table 3). Three journals accounted for 44% of the sampled articles: Clinical Cancer Research (17.5%), Cancer Chemotherapy and Pharmacology (13.6%) and Journal of Clinical oncology (12.6%). In all but one (based on an accelerated titration scheme) of the 103 reports, the ‘3 + 3’ design was used.

The 15 items selected in our checklist are summarized in Table 2 along with the percentage of reporting. The median credit sum over the 15 items was 10 (range: 6–13), with no difference according to the journal (p=0.49). Four items were reported in more than 95% of papers: Identifying the MTD or the RPTD as the main objective and scoring toxicities on international grading systems were both reported in 98 (95.1%) papers, reporting the starting dose in 102 (99.0%) and the number of dose levels in 100 (97.1%). By contrast, two items seemed to be outliers. ‘Was the estimated MTD or the RPTD associated with a measure of variability?’ was never reported. The second apparent outlier was ‘The first level was justified from preclinical or clinical data,’ which was reported in 28.2% of papers.

Discussion and Conclusions

Phase I trial papers carry significant information for drug development, particularly in oncology, in which clinical trial subjects exposed to an investigational cytotoxic agent are actual consenting cancer patients and not healthy volunteers. Thus, biased results from poorly designed and poorly reported trials could mislead decision-making in drug development affecting both individual patients and subsequent clinical trials. However, critical appraisal of the quality of phase I trials would be possible only if the design, conduct, and analysis of these trials are thoroughly and accurately described in published articles. This can only be achieved through authors’ complete transparency from authors, as exemplified in the CONSORT statement for reporting randomized trials [16]. Thus, quality assessment measures are mandatory. Nevertheless, although many reviews have documented deficiencies in the reporting of clinical trials, there has been less attention placed on quality assessment criteria for reporting on phase I trials and in phase I oncology trials in particular [11,12]. Therefore, we aimed at deriving a qualitative checklist for reporting phase I cancer trials – offering as a guideline the minimum elements that should be reported in all phase I trials to offer a consistent standard to inform readers and to inform further clinical drug development, but recognizing a full and thorough published report would also include additional elements as well.

Based on 15 items, our proposed questionnaire aims to assist in the accurate interpretation of the reporting of a trial’s objective, design, and analysis. Recently, Strevel et al. published a related checklist to assess the quality of abstract reporting for phase I cancer trials [24]. Strevel and his co-authors surveyed 27 experts and derived 22 elements of phase I trials to include in their checklist of abstract reporting criteria, of which nine were close to those specified in our own checklist of 15 elements (that is 60% of our items). Of these nine items in common with our phase I trial reporting checklist, three items related to data analysis: (1) the number of clinical trial subjects, (reported in 72.8% of papers vs. 30.8% of abstracts); (2) the dose escalation method (reported in 45.6% of papers vs. 13.8% of abstracts); (3) the conclusion of the study with regards to the MTD or RPTD, (reported in 56.3% of papers vs. 36.4% of abstracts).

Two items appeared rather poorly reported in published articles, namely the justification of doses and the report of variability of the estimated MTD. However, the former had an average score of only 1.35, and while 91% of participants aggreed with the inclusion of this item in the questionnaire, only 53.5% strongly agreed. The latter could appear conflicting, pointing out the opposition between rule-driven and model-driven methods, as suggested by one expert (AI). Indeed, model assumption will help narrow down the variability. By contrast, the quality of MTD estimation in rule driven methods depends on dose-toxicity relationship and is hard to access and report, as observed in our survey. However, our aim was to derive some checklist to promote standardized reporting for phase I trials, so that accurate interpretation of results from these trials could be reached. Thus, even based on standard ‘3+3’ designs, the problem should be stated as an estimation problem, to provide meaningful information for future trials. Although, statistical considerations have been overlooked in these studies until the 1990’s, this points out the need for dissemination of the innovative methods that address these issues [25–27]. Many authors have shown that on average the dose selected by the ‘3+3’ design has a DLT rate of about 0.20 [28–30], so that the MTD could be presented as the dose for which the probability of DLT is 0.20. Moreover, improved precision can be reached by using a generalization of the ‘3+3’ design, denoted the ‘A+B’ designs [31], with cohort sizes A or B larger than 3. Such designs can also be useful if the MTD is defined as a dose with DLT rates other than 0.20. Maximum likelihood estimators have been derived under a two-parameter logistic distribution [32] so that these designs could appear as intermediate between fully rule- and model-based approaches.

Some limitations of our study should be discussed, notably the number of actual physicians and statisticians whom we surveyed out of the potential respondents in the survey pool to whom emails were sent asking for their participation. Nevertheless, the number of survey respondents in our study greatly exceeded those sampled by three other studies (N<30 for each of those studies) that also evaluated quality assessment measures for reporting phase III clinical trials [13,21] or phase I trials [24].

In summary, empirical research concerning assessment of the methodological quality of phase I trial design, conduct, analysis and reporting is relatively new. Our study proposes a generic list of minimum criteria for quality assessment in phase I oncology trial reporting that may be a first step toward a minimum reference standard of quality measures for all phase I trials. The validity of these criteria list will have to be measured and evaluated over time. Otherwise, we hope that checklists or guidelines may act as a vehicle to improve phase I reporting and therein the quality of phase I trials. Indeed, such a list could facilitate the initial design and protocol writing of a phase I trial. Finally, owing to the growing development of targeted therapy in oncology, there will be further need to address the specificities of trials testing noncytotoxic agents in such quality assessment scales.

Appendix

Initial questions of the survey to which study participants responded. ‘x’ refers to the item selected by the quantitative analysis using three criteria (1 = in strong agreement for inclusion, 2 = in agreement for inclusion (strong or moderate), and 3 = mean rank score of each item) which generated sufficient consensus (Q3 cut-off as the threshold) among survey respondents. Meeting of these thresholds was further divided among all survey respondents (clinicians and statisticians = ‘Overall’) and the subset of statisticians alone (‘Stat.’).

Questions	Criteria
	Overall			Stat.
	1	2	3
Was the study designed for dose-finding?	x	x	x	x
Was the objective to find a MTD?	x	x	x	x
Was the MTD clearly defined?	x	x	x	x
Was the objective to find a DLT?
Did the objective include a Pharmaco-kinetics analysis?
Was the study mono or multi-centric?
If it was multi-centric, was the study national or international?
Was the experimental treatment a combination of several therapeutic agents or only one?
Was the dose allocation method described?	x	x	x	x
If yes, was it justified?		x	x	x
Was the starting dose specified?	x	x	x	x
If yes, was it justified?		x
Was the number of dose-levels mentioned?		x	x	x
Were dose levels determined from preliminary animal or human studies?
Was the dose levels specification method given?
If yes, was it justified?
Was a prior dose-toxicity relationship described before the trial?
Was a non-decreasing dose-response relationship assumed?
Did the determination of dose meet ethical considerations?	x	x	x	x
Was the outcome assessor blinded?
Was the patient blinded?
Was the DLT precisely defined?
If yes, was it based on international grading systems such as OMS or NCI scales?	x	x	x
Was only the toxicity supposed related to the drug used to define the DLT?
Was the MTD defined?	x	x	x	x
Was the sample size fixed at the beginning of the trial?
Was the sample size justified?
If yes, was it adequate?
Was the initial fixed sample size reached?
If no, was any justification given?
Was the period of enrollment mentioned?
Did the study include patients with different diseases?
Was the patient population specified through the inclusion/exclusion criteria?	x	x	x	x
Was the MTD determined?	x	x	x	x
Was the probability of toxicity associated with the MTD given?
Was a precision given for the estimated MTD?
Were patients exposed to too low or too high dose?
Did the dose allocation along the trial match the described method?				x
Did the dose levels match the specification method?
Was the frequency of adverse effects acceptable?
Was treatment Pharmaco-kinetics profile estimated?
Was the efficacy evaluated?
Did efficacy modelization take into account for the dose allocation procedure?
Was there any statistical modeling of toxicity?
Were some patients withdrawn from the analysis?				x
Did the analysis include an intention-to-treat analysis?
Was the mortality reported?	x		x
If yes, was it related with the drug?	x