Methodological characteristics of academic clinical drug trials – a retrospective cohort study of applications to the Danish Medicines Agency 1993–2005

Abstract

AIM

The aim of this study was to investigate the temporal trends in characteristics of academic clinical drug trials. We here report characteristics on trial methodology.

METHODS

A review of 386 approved applications of academic clinical drug trials submitted to the Danish Medicines Agency 1993–2005 was carried out. Data on 11 methodological characteristics were collected, e.g. statement of primary endpoint, use of control group, blinding, randomization, method for generation of allocation sequence, monitoring according to the principles of Good Clinical Practice (GCP monitoring) and publication.

RESULTS

Statement of primary endpoint increased from 60 to 90% of trials (P < 0.0001). Comparing the period before and after implementation of the Clinical Trials Directive in 2004, intention of GCP monitoring increased from 13% to 94%. Control of medicine compliance increased from 42% to 76% (P < 0.0001) among trials with self-administration of the investigational medicinal product. Among controlled trials use of randomization increased from 78% to 94% (P= 0.0063) of trials. Remaining characteristics did not change significantly. In total 68% (264/386) were randomized controlled trials.

CONCLUSIONS

Our study shows that randomization, definition of primary endpoint, GCP monitoring, and control of medicine compliance form part of a significantly increasing percentage of academic clinical drug trials. This indicates an increase in the quality of academic clinical drug research in Denmark 1993–2005. However, high numbers of unblinded randomized controlled trials and randomized controlled trials utilizing unacceptable methods for generation of allocation sequence emphasize the potential for further improvement of trial methodology.

WHAT IS ALREADY KNOWN ABOUT THIS SUBJECT

• From 1993 to 2005, the number of academic clinical drug trials in Denmark decreased by 38% with a small increase in 2006. Commercial clinical drug trials in Denmark were subject to a similar decrease. Implementation of the European Clinical Trials Directive in 2004 had no immediate effect on the number of either academic or commercial clinical drug trials.

WHAT THIS STUDY ADDS

• Our study shows that randomization, definition of primary endpoint, monitoring according to the principles of Good Clinical Practice (GCP monitoring), and control of medicine compliance form part of a significantly increasing percentage of academic clinical drug trials. This indicates an increasing quality of these trials.

• However, high numbers of unblinded randomized controlled trials and randomized controlled trials utilizing unacceptable methods for generation of allocation sequence emphasize the potential for further improvement of the methodological quality.

Introduction

In the period 1993–2005 the number of academic clinical drug trial applications received by the Danish Medicines Agency showed a gradual decline from 147 to 86 applications per year [1]. The reason for this decline remains unknown although, contrary to the belief of many researchers, it seems independent of the introduction of the European Clinical Trials Directive in 2004. Implementation of the directive was a major legislative change establishing standardized application and safety reporting routines as well as imposing adherence to Good Clinical Practice (GCP).

The underlying purpose of academic and commercial drug trials differ. Whereas commercial drug trials are often conducted in order to gain data required for development and marketing purposes, academic drug trials tend to address various scientific questions regardless of whether a profit can be made. It is therefore crucial to maintain a reasonable number of good quality academic as well as commercial trials and, accordingly, the continuous decrease in the number of academic drug trials seems critical. However, as a small number of good quality clinical trials may produce better evidence than a large number of poor quality trials, further knowledge about changes in the quality of these trials is needed.

The aim of this study was to investigate temporal trends in the characteristics of academic clinical drug trials. We here report characteristics of trial methodology extracted from approved applications for academic clinical drug trials (protocols and correspondence) submitted to the Danish Medicines Agency 1993–2005.

Methods

The study was performed as a retrospective cohort study of a subset of approved applications of academic clinical drug trials submitted to the Danish Medicines Agency in 1993, 1995, 1997, 1999, 2001, 2003, 2004 and 2005. Identification of the applications is described by Berendt et al. [1]. Applications were listed according to year of submission. From each selected year 1993–2003 approximately 45 applications (range 45–47) were selected at random using a computer generated list. Lack of data precluded a formal power calculation. Even distribution (odd years 1993–2003) was selected to ensure a sufficiently detailed description of any temporal changes while maintaining a reasonable sample for each data point. Due to the short time frame from implementation of the Clinical Trial Directive May 1 2004 to data collection July–September 2006 all academic clinical drug trial applications submitted May 1 2004 to December 31 2005 were included in the study.

Approval by the competent authority may require alterations of the submitted protocol. Therefore, data for each trial were collected from all relevant material available at the Danish Medicines Agency from submission to approval.

Selection of variables

In the absence of a generally accepted method for assessing the quality of clinical trials, 11 methodological characteristics were selected as quality indicators (Table 1). The selection was inspired by:

Table 1.

Selected characteristics associated with trial, protocol, or publication quality according to the CONSORT Statement [2], ICH guidelines E8 and E9 on clinical trials [3, 4], Jüni et al. [5] and the Danish Medicines Agency guideline on application for authorization of clinical trials [6]

	This study	CONSORT statement	ICH	Jüni	DKMA†
Clearly defined primary and secondary outcome measures	X	X	X		X
Statistical power/calculation of sample size	X	X	X		X
Statistical methods for data analysis	X	X	X		X
Controlled/not controlled trial	X	–*	X		X
Parallel/crossover design	X	–*			X
Randomization	X	–*	X		X
Method used to generate the random allocation sequence	X	X	X
Blinding	X	X	X	X	X
GCP monitoring	X		X		X
Medicine compliance measures	X				X
Publication	X				X
Concealment of treatment allocation		X		X
Handling of withdrawals and dropouts		X	X	X	X

^*The CONSORT statement applies to parallel design RCTs only.

^†Danish Medicines Agency.

1. the CONSORT statement [2],

2. International Conference of Harmonization (ICH) Topic E8 [3] and E9 [4] on general and statistical considerations for clinical trials, respectively,

3. a study on quality scoring of clinical trials by Jüni et al. [5],

4. the Danish Medicines Agency guideline for application for authorization of clinical trials of medicinal products on humans [6], and

5. an initial analysis of the content of the earliest protocols.

Concealment of treatment allocation and handling of withdrawals and dropouts were not included due to absence of information in the earliest protocols.

Data collection

All applications were reviewed with regard to the following characteristics:

Statistics

We determined whether primary endpoint(s), statistical power and statistical methods for data analysis were stated.

Controlled/uncontrolled trials

We assessed whether the trial comprised at least one control group.

Good Clinical Practice (GCP)

Adherence to GCP was assessed on the basis of intended monitoring and medicine compliance control, the latter only if relevant (e.g. if study medication was administered by the subjects themselves).

Publication

Publication rate was estimated by searching Medline July–September 2006 for investigators listed in the applications in combination with drug names (preferably by MeSH-term). To determine possible associations, the identified abstracts were compared with the protocols. Case reports were excluded.

Controlled trials and randomized controlled trials were subject to further investigation:

Controlled trials

Type of control group(s), presence and type of blinding, randomization and design (parallel vs. crossover) were determined.

Randomized controlled trials (RCTs)

We evaluated the method for generating the allocation sequence. Acceptable methods included predetermined lists, such as computer generated lists of random numbers, statistical tables and randomization performed by external, statistically competent professionals. Methods such as roll of the dice or use of even/uneven dates were considered unacceptable methods.

Data analysis and statistical procedures

Data were analyzed in SAS/JMP 6.0.2 using logistic regression, except intention of GCP monitoring and medicine compliance which was analyzed using Fisher's exact test and variables showing poor log-linear relationship (P_{lack of fit} < 0.05) which were analyzed with χ². P values < 0.05 were considered statistically significant. The binomial variance in the logistic regression model takes the sampling into account. As data from 2004 and 2005 constitute full populations from the respective periods, confidence intervals (CI) should be interpreted with caution.

Inter-rater reliability between LB and CH was assessed from a set of applications selected at random. Interpreting the kappa values according to Landis et al. [7], six of the eleven characteristics showed perfect agreement (κ= 1), four demonstrated substantial agreement (κ range 0.62–0.71) and one (publication) showed moderate agreement (κ= 0.52).

Results

Overall results

The results are based on application material from 386 approved drug trial applications comprising 53% (386/730) of academic clinical drug trial applications submitted in 1993, 1995, 1997, 1999, 2001, 2003, 2004 (after May 1) and 2005.

Figure 1 shows the percentage of trials with a defined primary endpoint and of trials for which GCP monitoring was intended. Figure 2 shows percentage of randomized trials in the subgroup of controlled trials. The occurrence of these characteristics increased significantly during the period under study. Figure 3 shows the publication rate. The remaining characteristics did not change significantly (Table 1).

Figure 1.

Percentage of academic drug trials with a defined primary endpoint (♦, P < 0.0001) and percentage of academic drug trials in which monitoring was intended (▵, P < 0.0001 for 1993–2003 compared with 2004–05, Fisher's exact test)

Figure 2.

Percentage of randomized trials (P= 0.0063, subgroup: controlled academic drug trials)

Figure 3.

Publications by year of submission of application. Only data on academic clinical drug trials with expected end of trial prior to 1 July 2006 are shown

Characteristics of controlled trials

Controlled trials comprised 79% (304/386) of the trials (Table 2). Of these, 110 (36%) were controlled only by placebo, 76 (25%) only by active comparator(s) and 34 (11%) by both active comparator(s) and placebo. Other types of control groups, e.g. historical control groups or non-medicinal interventions such as exercise or diet, were used in 84 (28%) of controlled trials (Table 2). Temporal trends in the use of control groups are shown in Figure 4.

Table 2.

Trial characteristics with no significant changes over time

	n		95% CI*	P
Academic drug trials	386	–	–	–
Calculation of statistical power
Yes	277	72%	67.1, 76.0	†
No	109	28%	24.0, 32.9	†
Statement of statistical methods for data analysis
Yes	294	76%	71.7, 80.1	†
No	92	24%	19.9, 28.3	†

	n		95% CI	P
Subgroup: controlled trials	304	79%	74.4. 82.5	0.11
Control group combinations
Active comparator(s) and placebo	34	11%	8.1, 15.2	0.26
Only active comparator(s)	76	25%	20.5, 30.2	0.37
Only placebo	110	36%	31.0, 41.7	0.76
Other	84	28%	22.9, 32.9	0.69
Blinding
Double-blind	163	54%	48.0, 59.1	0.88
Single-blind	17	6%	3.5, 8.8	0.24
Open label	123	40%	35.1, 46.1	0.60
Design
Parallel groups	209	69%	63.0, 73.7	0.69
Cross over	92	30%	25.4, 35.6	0.42
Other	3	1%	–	–

	n		95% CI	P
Subgroup: randomized controlled trials (RCTs)	264	68%	63.6, 72.8	0.78
Generation of allocation sequence
Acceptable method	114	43%	37.3, 49.2	†
Not acceptable method	54	20%	16.0, 25.7	0.74
Method not stated	96	36%	30.8, 42.3	†

^*CI: confidence interval. P values were calculated using logistic regression.

^†Poor log-linear relationship (P_{lack of fit} < 0.05), but P_Chi < 0.05 indicating significant differences between years.

Figure 4.

Types of control groups used in controlled trials. Percentage of trials using only placebo (—), percentage of trials using only active comparator(s) (---), percentage of trials using both active comparator and placebo (-- - --) and percentage of trials using other types or combinations of control groups (grey line)

Of the 304 controlled trials 163 (54%) were double-blind and 17 (6%) were single-blind. The remaining 123 (40%) were unblinded (Table 2). A parallel design was used in 209 (69%) of the controlled trials while 92 (30%) used a cross-over design (Table 2). Randomization was utilized in 41/45 trials in 1993 and 61/68 trials in 2005 suggesting a significant increase from 78% (95% CI 66, 91) to 94% (95% CI 88, 100) of controlled trials (P= 0.0063, Figure 2). An acceptable method for generation of the allocation sequence was stated for 114 (43%) of the 264 RCTs, while 54 (20%) stated an unacceptable method. The remaining 96 (36%) trials lacked statement of method (Table 2). A log-linear relationship could not be established for either percentage of RCTs stating an acceptable method or percentage of RCTs lacking statement of method.

Statistical characteristics

A primary endpoint was stated for a significantly increasing percentage of trials, as 32/45 of trials stated a primary endpoint in 1993 compared with 82/84 in 2005 suggesting an increase from 59% (95% CI 44, 73) to 92% (95% CI 87, 98, P < 0.0001, Figure 1).

For 277 (72%) of the 386 trials, the number of subjects was substantiated by calculations of statistical power (Table 2). Methods for statistical analysis were stated for 294 (76%) of the 386 trials (Table 2).

GCP and GCP monitoring

The share of trials in which GCP monitoring was intended increased significantly from 2/45 in 1993 to 82/84 in 2005. Even though the major part of this increase was observed from 2003 to 2004, the share of trials in which monitoring was intended increased significantly in the period 1993–2003 as well (P < 0.0003, Figure 1). Compliance control measures were stated for 42% (52/125, 95% CI 33.3, 50.4) of trials submitted before 2004 for which compliance control was relevant (self-administration of investigational medicinal product by subject). In 2004 and 2005, the share had increased significantly to 76% (47/62, 95% CI 63.8, 84.7, P < 0.0001, Fisher's exact test, data not shown).

Publication

Publications were identified for 74 (33%) of 227 approved trials submitted 1993–2001 with expected end of trial prior to July 1 2006 (Figure 3). The publication rate did not change significantly among trials submitted in the period 1993 to 2001. Among trials submitted after 2001 the publication rate decreased gradually to 0% of trials submitted in 2004 and 2005 indicating a publication lag time of at least 2.5 years.

Discussion

Summary

In our study of approved protocols of academic drug trials we found that the use of randomisation in controlled trials and definition of a primary endpoint as well as adherence to GCP increased significantly from 1993 to 2005. This indicates an increasing quality of academic research. The share of trials that are published seems stable. A number of methodologically important characteristics, such as use of blinding, control group and unacceptable methods used for generation of allocation sequence, did not change significantly.

New methodology

To our knowledge this is the first study of temporal trends in methodological characteristics extracted from protocols of academic drug trials. In the absence of comparable studies, data on similar characteristics extracted from other types of trials are used for comparison where available. Hence, our finding of statement of statistical power in 72% of the protocols is consistent with the percentage of 76% found by Soares et al. [8] among academic oncology trials in the UK.

Randomization is a well-established method for reducing selection bias [9, 10]. Our finding of 20% unrandomized trials among 304 controlled trials is therefore unacceptably high. The fact that 40% of the controlled, randomized studies (RCTs) are performed without blinding is also disturbing, as Schulz et al. [11] have demonstrated that trials that are not double-blind yield significantly larger estimates of treatment effects. Although the nature of certain comparators precludes blinding, this hardly explains all 40%.

Characteristics regarding concealment of allocation and handling of withdrawals and drop-outs are correlated to effect size and, hence, would have been relevant to include in this study. However, data from 20 protocols extracted as a pilot study did not provide information on these characteristics and they were not collected for the remainder of the protocols.

The publication rate of 33% of academic trial protocols approved in Denmark is comparable with publication rates of 29% in France among approved biomedical research projects [12] and somewhat higher than 19% found in Spain among approved clinical trial protocols [13]. However, the latter two publication rates include both academic and commercial trials. The percentage of completed Danish trials is assumed equal to or higher than the publication rate. The exact percentage remains unknown.

What is the net effect of increasing quality of a decreasing number of trials?

The results of our study should be seen in conjunction with the gradually decreasing number of trials [1]. We speculate that the temporal trend in absolute terms is either

1. A constant number of trials with methodological characteristics associated with good quality and a decreasing number of other trials, or

2. An increasing number of trials with methodological characteristics associated with good quality and a decreasing number of other trials.

The latter scenario is most desirable as it indicates a shift from trials of poor methodological quality to good methodological quality. The overall scientific impact of the declining number of academic drug trials in combination with improvements in methodological characteristics, however, remains unknown.

What is quality of clinical drug trials?

In relation to the above reflections, it should be kept in mind that quality of clinical trials is still to be defined. However, a substantial number of quality scales and checklists have been designed, the majority consisting of a mix of methodological and reporting characteristics [14]. Using a different approach our study incorporates methodological characteristics in combination with potential of protocol compliance (adherence to GCP). As discussed by Gögenur & Rosenberg [15], adherence to GCP does not per se ensure the quality of a trial. However, it provides means of controlling and assuring the validity of the collected data, thus facilitating compliance with the research protocol. This is crucial, since even the best design cannot compensate for erroneous data handling. Accordingly, quality assessment of clinical drug trials should incorporate more than solely methodological and reporting characteristics, but should also include relevance, protocol compliance and diffusion of knowledge.

Protocols vs. publications as data source

Since research protocols are rarely obtainable, many quality assessment reports use publications as data material. However, observed median lag times of 5.5 years from start of enrolment to publication and 2.4 years from completion to publication [16] makes publications unsuitable as material for the evaluation of recent changes, e.g. caused by changes in legislation. To circumvent this potential lag time bias, we reviewed approved research protocols instead of publications, thereby providing a more concurrent estimate of temporal trends in academic research. The applicability of this method is partly supported by data published by Soares et al. [8], which show that the actual conduct of 56 academic oncology trials demonstrated better agreement with the research protocols than with the publications. However, protocols as well as publications are surrogate measures of the true conduct of a given study.

As we had unlimited access to all drug trial applications submitted to the Danish Medicines Agency 1993–2005, the selection bias associated with use of data from publications was prevented.

We are aware of four specifics of the Danish system that should be taken into consideration when applying our results to other European Union member states. (i) During the whole period under study, it has been mandatory in Denmark to obtain approval from both a regional ethics committee and the Danish Medicines Agency. (ii) In 1995, the first regional public GCP Unit was established providing GCP monitoring and guidance to researchers conducting academic drug trials. In 2003, two additional regional units were established thereby making the service available throughout the country. (iii) Measured by number of publications of randomized controlled drug trials per million inhabitants or by number of graduate students in life sciences, Denmark has been shown to have a high publication output compared with other European Union member states [17]. (iv) The percentage of academic drug trials in Denmark is high relative to other European member states (31.1% of trial applications registered in EudraCT 2004–09 vs. 20.5% in EU). We speculate that while the two former specifics may help increase focus on the quality of the protocol, the influences of the two latter are unclear. Hence, a high publication output may indicate a high research activity. However, a high research activity may indicate a higher quality of the studies as the researchers are experienced, or it may indicate a lower quality as scattering of resources limits those available for each trial. Nevertheless, the above is merely speculation that needs further investigation.

Studies on possible differences in trial characteristics between uncompleted and completed trials are recommended as well as studies on reasons for publication/non-publication.

To investigate the long term effects of the European Clinical Trials Directive as well as future changes potentially affecting the drug research environment, we recommend evaluation of selected characteristics of academic, as well as commercial, drug trials at regular intervals.

In conclusion, although the use of randomization and definition of a primary endpoint in controlled trials as well as adherence to GCP improved from 1993 to 2005, the results of our study emphasize the potential for further enhancement of clinical trial methodology in academic drug research. This should be attained by improving pregraduate education in clinical trial methodology and increasing the control with methodological aspects during the approval process of clinical trials at competent authorities.

Competing interests

There are no competing interests to declare.

Ethical approval

Not required.

Contributors

LB and CH retrieved and analyzed the data. KFB, KD, and PBA helped plan the study. LGP helped retrieve the data and plan the study. EA supervised and co-ordinated the study and helped retrieve the data. HEP had the original idea for the study and helped plan the study. LB and KD drafted the paper. All authors contributed to the final writing process and data interpretation.