Why Do So Many Drugs for Alzheimer's Disease Fail in Development? Time for New Methods and New Practices?

Abstract

Alzheimer's disease (AD) drug developments and clinical trials (CT) remain vulnerable to problems that undermine research validity. Investigations of CT methods reveal how numerous factors decrease active drug-placebo group differences and increase variance, thereby reducing power to reach statistical significance for outcome measure differences in AD CTs. Such factors include, amongst many, inaccuracy, imprecision, bias, failures to follow or lack of operational protocols for applying CT methods, inter-site variance, lack of homogeneous sampling using disorder criteria. After a review of the literature and survey of a sample of AD and Mild Cognitive Impairment (MCI) CTs, the authors question whether problems of human error preclude AD researchers from continuing their dependence on rated outcome measures for CTs. The authors propose that the realities of AD, especially a probable irreversible progression of neuropathology prior to onset of clinical symptoms or signs capable of differentiating persons at risk for AD from normal aged, require AD investigators and clinicians to privilege biomarkers and encourage their development as surrogate targets for preventive AD treatment developments, testing, and use in clinical practice.

Introduction

According to recent research into clinical trial (CT) methodologies, Alzheimer's disease (AD) drug developments and CTs remain vulnerable to lurking problems capable of compromising future research. To determine whether or not research into limitations in CT methodologies are taken into consideration in AD drug developments at present, for the current article we reviewed the literature that we could find with potential relevance for AD CT and drug development problems, developed a list of concerns with potential problems from that review, and then, using this list of potential concerns, we surveyed a representative sample of past and current AD drug development publications for evidence that investigators demonstrated awareness of methodological risks to the validity of their studies.

Four drugs are approved and currently used in AD: donepezil (Aricept) 1997, rivastigmine (Exelon) 2000, galantamine (Reminyl) 2001, and memantine (Namenda) 2003. In contrast, we recently identified over 100 compounds tested as potential therapies that were either abandoned in development or failed in CTs [1]. Furthermore, cholinesterase inhibitors (ChEIs), approved for AD, lately failed in CTs to prevent progression of patients with Mild Cognitive Impairment (MCI) to AD. Currently, many new compounds are in clinical testing or will shortly enter clinical testing for AD and MCI [2]. Our concerns are that methodological factors, relatively widely reported in the literature as barriers to CT successes, will interfere with investigators providing a fair test for these new AD drug candidates [1, 3]. In this paper we consider our own experiences with AD drug development, published reports on CT difficulties especially in neurology, psychiatry, and AD, and we review 40 AD drug developments randomly selected from known AD drug candidates in order to judge whether or not past problems in AD drug development may unnecessarily impede research into the efficacy of drugs either now in CTs or about to enter clinical development.

Since the 1984 publication of NINCDS-ADRDA criteria for diagnosing AD, progress understanding relevant genetics, imaging, biochemistry, and clinical presentations justifies revisions to AD diagnostic criteria [4]. Recently, Pangalos et al. [5] described advances in drug development practices over the last two decades. The potential importance of these technological advances is suggested by the latest Institute of Medicine [6] review of studies used to formulate clinical policies for treatment of post traumatic stress disorder (PTSD) [7]. The IOM Committee on Treatment of Post Traumatic Stress Disorder found 2771 studies of PTSD but evaluated only 90 as sufficiently sound scientifically to include in their review. They found the evidence inadequate in these reviewed studies to support the efficacy of any psychoactive drugs or psychotherapies for PTSD other than exposure therapy. The IOM Committee cited numerous problems with sample size, blinding, independence of investigators, the need to exclude studies that were case studies and case series, high dropout rates capable of introducing bias, influences associated with pharmaceutical funding, publication bias, population selections, and other factors [6, pp 124-125, 136). After ascribing some problems to earlier studies not using current investigational standards, to overcome the “scientific inadequacy” of available CTs in PTSD, the IOM report recommends that investigators use “the latest and most rigorous methods for designing and executing study protocols”. A Type II error occurs with the mistaken dismissal of a research hypothesis of drug efficacy because study conditions failed to provide a fair opportunity for the drug to express its true effects [3]. In the current paper, we ask whether, by currently taking advantage of research into methodological limitations that can lead to Type II errors in interpreting research studies, AD drug development exemplifies the progress in clinical research expressed in the IOM report.

Background

Overview of concerns with AD drug development

Currently, investigators focus AD and MCI investigations on disease modification rather than symptom remission. In AD drug development, the CT, originally designed for short-term evaluations of treatments, has not yet been successfully modified to give investigators confidence that disease modifying drug effects can be successfully tested [8-10]. Long duration trials with non-parallel initiations and terminations of treatment to detect disease modifying drug effects risk increased losses of subjects and other compromises to their integrity [11]. It is widely accepted that designs able to test disease modifying drug effects will require larger numbers of subjects, greater numbers of clinical research sites, and longer double-blind evaluations [12]. Under these conditions, inter-site variance, difficulties monitoring sites for conformity to protocols, increased losses of subjects, placebo responses, and life-events more easily undermine CTs [13, 14].

Pangalos et al. [5] point out that the rate of AD drugs coming to market falls below the already low 7% rate for all central nervous system drugs. Consistent with the IOM report on development of treatments for PTSD and with Pangalos et al., we and other authors raised, for clinical phases of neurological, psychiatric, and AD drug developments, serious questions about the reliabilities of development methodologies and practices [1, 15]. As Esterbrook et al. [16] and others encountered as they attempted to evaluate development programs for drugs, many unsuccessful preclinical and clinical studies are not reported in the literature and results will not be released by sponsors [17]. Occasionally, a synopsis that lacks considerable detail may appear as a press release [1]. These long-recognized problems exerts an unknown effect on our understanding of how methodologies and practices may contribute to drug development failures in AD [1, 5, 18].

Potential weaknesses affecting the validity of AD drug developments

Investigators find problems of inaccuracy, imprecision, and bias interfering with CTs that are dependent on using clinical ratings as outcome measures [14, 15, 19-25]. Error components, added to clinical ratings by inaccuracies, imprecision, failures to follow or lack of operational protocols for applying methods, and bias, have been shown to decrease active drug-placebo group differences and to increase the variance in data. These effects on data reduce the possibilities for reaching statistical significance for outcome measure differences [3, 15]. Similar measurement errors and a lack of specificity during diagnostic evaluations and qualifications of subjects for eligibility for a CT can include subjects incapable of responding to treatment because of misdiagnosis, genetics, or specific pathology [25-26]. Each of these factors reduces the power of the CT to detect drug-placebo differences; requiring additional subjects, additional sites to provide subjects, more investments in training investigators, risks of recruiting clinically inexperienced investigators, increased resources for monitoring sites thus contributing further risks of compromising the integrity of the CT. Studies have shown how inaccuracy and imprecision in outcome ratings cause CTs to fail from compromises to power and from complications that follow from additions of subjects to increase power [3, 12, 24, 25, 27-30]. Unreliability, due to limitations in the sensitivity of outcome measures to change and human errors, risks false-negative reports and patients being exposed to inadequately designed and controlled research conditions [1, 31, 32].

Problems with raters

Engelhardt et al. [15] found that over 50% of clinical evaluators in two multicenter trials of antidepressant drugs functioned poorly or only fairly as evaluators of outcomes. Cogger [27] demonstrated that these poor and fair raters reported no active drug-placebo group differences. Rather, only good and excellent raters, in terms of compliance with protocols for using the Hamilton Depression Scale, contributed to the effect size associated with the antidepressants [27]. Kobak et al.[33] determined that among 29 raters at 12 antidepressant study research sites 72% first learned to administer the Hamilton Depression Scale [34] in preparation for the investigation and that only 38% were observed using the scale with patients. Demitrack et al. [35] found excessive variance among sites in large multisite studies. Kobak et al. [36, 37] and Targum [38] showed beneficial effects of more intensive training of investigators at research sites. On the other hand, Demitrack et al. [35] and Kobak et al. [37] found that, for some prospective raters, extended training did not improve the reliability of their performances necessitating their exclusion from participation in CT research. There is no agreement over how to overcome problems of rater unreliability although, as a result of recent studies documenting the problems, investigators are debating means to overcome these limitations to CT validity [39, 40].

Problems of inter-site variance

Consistent with these concerns raised by researchers into CT methods, two highly experienced developers of AD drugs, Paul Aisen and Bruno Vellas, recently acknowledged the seeming inevitability in AD studies of “large degrees of variance amongst sites,” the large number of subjects requiring “70-80 sites” dispersed “around the world,” and sites contributing only “2-5 patients.” [12]. Cummings [41] claims that clinical evaluations using rating scales can be used as biomarkers and qualified as surrogate measures for AD disease severity. We and others present evidence of problems introduced by excessive variance from the human diversity of subjects, inherent imprecision in rated scale values, and shortcomings of evaluators prone to human errors and bias. This evidence suggests that clinical evaluations are at best too imprecise and inaccurate to overcome the need for large numbers of subjects to power CTs and the resulting complications from large numbers of subjects noted by Aisen and Vellas.

Problems with rating scales

AD rating scales are limited to ordinal data. Ordinal data express a sorting of subjects in relation to others or selves at different times; but, without any specific intervals expressed in the data. Unlike a change from 7 to 8 centimeters where the size of the interval is specified, for a subject to remember 8 rather than 7 words implies no specific quantity that separates the two categories. Rating scales can indicate but not quantify differences. As Winbald et al. [42] worry, “small numerical changes on scales…might…be clinically meaningful in practice” and vice versa. Effect size estimates, because they use qualitative and not quantitative data, do not entirely overcome these concerns. Recently investigators ascribed to time effects on ratings trends towards smaller rates of annual decline in AD patient groups demonstrating the latent instability of qualitative rankings [11]. The movement towards disease modifying interventions requires early diagnosis and evaluations of clinical status when only initial cognitive and behavioral effects from AD are manifest and possibly even earlier. Existing AD ratings scales exhibit ceiling insensitivities when used with pre-AD conditions. These insensitivities lead investigators to turn to more sensitive outcome measures such as the Neuropsychological Test Battery [31]. As we have noted [1], many rating scales and tests are insensitive to changes in early or pre-AD. Using these scales minor rated differences in pre-AD may not reflect much greater and irreversible changes in underlying pathology. This insensitivity to irreversible pathology suggests the importance of quantitative biochemical markers for identifying pre-AD as a disease process. Ordinal values obtained using rating scales are associated with relatively large inherent variances. This variance expresses imprecision in measurements and limitations when raters attempt to quantify changes in a patient's clinical state. Because of these limitations ordinal data from rating scales can not provide reliably sound grounding for extrapolating from rated clinical changes to neuropathological changes, especially in pre-AD states. AD preventive efforts, to be effective, may have to be applied prior to appearance of clinical disturbances creating serious evaluation problems for researchers and management problems for practitioners without reliable methods of clinical assessments for pre-AD.

Becker and Greig [1] and Becker [3] discussed how CT methodologies are made more problematic by the dependence of AD drug developments on rated clinical outcome measures. Estimates of sample sizes from studies of biomarker changes over the course of AD put into dramatic perspective the magnitude of the complications introduced into CTs by dependence on human judgments and rated outcomes [11]. Thal et al. [11] estimated that numbers of subjects needed per group in AD CTs could be reduced from the current range of 270-6377 (Table II) to 16 to 40 based on what is currently known of imaging and biochemical markers.

Table II.

Table II:1. Drug, Mechanisms, Pharmacological Activities, Design, and Publicationa,b
Methodological Query Is there discussion of…? Or relevant data reported about…?	Literature References	Approved AD Studies % Compliant	Failed AD Studies % Compliant	MCI Studies % Compliant	AD Drugs in Development Studies % Compliant
1. Why mechanism is investigated*	[Authors]	90%	39%	70%	50%
2. Drug Class Actions	[Authors]	70%	39%	70%	50%
3. Class Efficacy Evidence	“expect… with previously failed drugs with mechanism, … specific justifications for why this compound … a possible exception.” [Authors]	90%	39%	70%	50%
4. Why Drug Molecule was Chosen *	“independence of investigators, pharmaceutical funding, publication bias, [6, pp 124-125, 136]	100%	39%	70%	50%
5. Brain Drug Target Concentration Evidence*	Importance of animal studies for dosing to provide appropriate concentrations of drug at brain targets [1, 5]	0%	11%	10%	10%
6. Use of Microdosing Studies	Confirm drug concentrations at brain targets [1, 5]	0%	0%	0%	0%
7. Other PK evidence considered?	“drug kinetics and dynamics critical to effective use of metrifonate” [1]	0%	6%	10%	10%
8. Elderly PK/PD effects considered?	“changes with aging that potentially affect drug concentrations at molecular targets” [Authors]	0%	6%	10%	10%
9. Design Disease Modifying*/Symptomatic/Not an Efficacy study (%)	“disease modifying drug effect designs untested” [8-10]	100% 0/100/0	39% 1/1/0	100% 100/0/0%	100% 70/20/0% (One study no information)
10. Statistical analysis ITT specified	“FDA standard” [64]	100%	100%	100%	100%
11. Percent of Studies in Category that are published	“unsuccessful preclinical and clinical developmental studies not published and sponsors refuse to release information about the development,” “lack of access to failed studies has unknown effects on understanding how methodologies and practices contribute to drug development failures” [1, 5, 16-18].	90%	22%1	70%	0% (CTs only registered, plans not published)

Table II: 2. Dosinga,b
Methodological Query Is there discussion of…? Or relevant data reported about…?	Literature References	Approved AD Studies	Failed AD Studies	MCI Studies	AD Drugs in Development Studies
12. Phase II Evidence	[Authors]	10%	6%	20%	10%
13. Dosing Optimized by studies?	[Authors]	0%	0%	10%	0%
14. Dosing Confirmed in CT?	[Authors]	0%	6%	20%	0%
15. Are Dosing Issues Discussed?*	[Authors]	20%	0%	10%	0%
16. Compliance Noted in for this Study?*	[Authors]	20%	11%	20% MEAN = 86/89	0%
17. Basis of Dosing increases during Study	[Authors]	90% 70% by protocol	39% 100% by protocol	100% 100% by protocol	20% 20% by protocol
18. Percentage of Subjects at Target Dose*	[Authors]	60% MEAN = 91%	11%	80% RANGE = O-70%	10%
19. Interference by Drug Tolerance *	[Authors]	20%	11%	0%	10%
20. Adverse Event Rate Drug/Placebo	[Authors]	80% MEAN = 76/69	6%	40% MEAN = 91/85	0%
21. Discontinuation Rate due to Adverse Events Drug/Placebo	[Authors]	80% MEAN = 18/16	22% MEAN = 11/6	30% MEAN = 15/7	0%

Table II; 3. Research Subjectsa,b
Methodological Query Is there discussion of…? Or relevant data reported about…?	Literature References	Approved AD Studies	Failed AD Studies	MCI Studies	AD Drugs in Development Studies
22. Consideration of Sample Homogeneity & Equivalence across Studies*	“Needed revisions to AD diagnostic criteria” [4],“population selections, and other factors” [6, pp 124-125, 136); “diagnostic eligibility vagaries introduce subjects incapable of responding to treatment because of misdiagnosis, genetics, or specific pathology” [25], “MCI criteria not stable across administrations” [55, 115]	100%	39%	100%	22%
23. Presence of other diagnoses*	“whether other diagnoses and medications lead to adverse event reports that suppress dosing increases?” [1]	30%	6%	10%	0%
24. Presence of other medications*	“whether other diagnoses and medications lead to adverse event reports that suppress dosing increases?” [1]	60% MEAN = 88% of subjects	6%	20%	0%
25. Imaging at Entry	“Dubois et al. [4] anticipate imaging becoming part of AD diagnostic criteria”	10%	0%	20%	30%
26. Imaging for Fronto-temporal Dementia (FTD) Exclusion*	“NINCDS Criteria do not exclude FTD” [26]	0%	0%	0%	0%
27. Imaging as Diagnostic Confirmation*	“Dubois et al. [4] anticipate imaging becoming part of AD diagnostic criteria”	0%	0%	0%	0%

Table II: 4. Outcome Measuresa,b
Methodological Query Is there discussion of…? Or relevant data reported about…??	Literature References	Approved AD Studies	Failed AD Studies	MCI Studies	AD Drugs in Development Studies
28. Outcome Measures Specified	[Authors]	100%	33%	100%	50%
29. Rater Qualifications*	“Kobak et al.[33] report that among 29 raters at 12 research sites 72% first learned to administer the Hamilton Depression Scale in preparation for the investigation and only 38% were observed using the scale with patients”	0%	0%	10%	0%
30. Rater Reliability Training*	“problems of inaccuracy, imprecision, and bias interfere with CTs dependent on clinical ratings of outcome measures” [1, 15, 21, 23-25, 39, 40].	0%	V	20%	0%
31. Ratings Accuracy Verification*	“inaccuracies decrease detected active drug-placebo group differences, undermine statistical significance, and reduce effect size” [15, 27].	0%	0%	0%	0%
32. Ratings Precision Verification*	“imprecision increases the variance in data, undermines statistical significance, and reduces effect size” [3, 15, 21, 23].	0%	0%	0%	0%
33. Documentation of Reliabilities Established during Training and Study	“Kobak et al. [33] and Targum [38] showed the effects of more intensive training of investigators at research sites.” “More reliable outcome measures can reduce sample size requirements.” [30]	0%	0%	10%	0%
34. Rater Bias in Placebo Improvements*	“bias, indicated by placebo group improvement, potentially decreases the detected active drug-placebo group differences and increases the variance in data” [3, 15]	0%	0%	0%	0%
35. Are there Rater Disqualifications?	Demitrack et al. [35] and Kobak et al. [37] found some raters can not qualify in training.	0%	0%	0%	0%
36. Are Means for Multiple Data Points Used for Statistical Analysis?*	“Means of data points reduce variance” [3, 21, 22]	0%	0%	0%	0%
37. Are Surrogate Outcome Measures (SOM) Used?	“Rating scale unreliability requires more reliable indicators” [1, 4, 62]	0%	6%	20%	60%
38. Are SOM Validated?	“Quantitative SOM are not yet available” [61, 61]	0%	0%	0%	10%

Table II: 5. Research Sites and Investigatorsa,b
Methodological Query Is there MENTION/discussion of…? Or relevant data reported about…?	Literature References	Approved AD Studies	Failed AD Studies	MCI Studies	AD Drugs in Development Studies
39. Study number (N) of subjects	“Size issues: [6, pp 124-125, 136). Aisen & Vellas acknowledged large number of subjects as problems” [12]	100% RANGE 93-632 MEAN = 391	39% RANGE 6-341 MEAN = 150	100%; RANGE 270-6377; MEAN = 1627	100%; RANGE 48-950; MEAN = 292
40. N % lost during study Drug/placebo*	“high dropout rates capable of introducing bias, (IOM 2007, pp 124-125, 136). Long-term disease modifying designs risk losses of subjects” [9, 10, 12-14]	100% MEAN 27/20	39% MEAN = 14/10	70% MEAN = 27/23	20%;
41. Number of Sites*	“Larger numbers of subjects require greater numbers of clinical research sites”. Paul Aisen and Bruno Vellas noted problems with large numbers of subjects requiring “70-80 sites” [12]	80% RANGE = 18-50 MEAN = 32	22% RANGE = 1-28 MEAN = 11	60% RANGE = 6-69 MEAN = 36	90%; RANGE = 1-67 MEAN= 21
42. No. of Sites < 6 subjects	“Aisen and Vellas acknowledged as problems sites contributing only 2-5 patients.” [12]. [Authors] “concerned that small sites inexperienced with AD”	0%	0%	0%	10%;
43. Documented site Diagnostic experience*	“Rater bias leads to lax applications of diagnostic criteria to secure subjects” [33]	0%	0%	0%	0%
44. Site Investigator Training Recorded*	“Conditions of research an essential aspect of effective clinical use of drug and evaluations of CT reports” [1]	0%	0%	0%	0%
45. Inter-site Variance Effects on Statistical Analysis Studied*	“inter-site variance undermines CTs power” [13, 14]. Acknowledged in Butcher [12] by Aisen & Vellas, “large degrees of variance amongst sites.” [35]	0%	0%	0%	0%
46. Site Performance in Earlier CTs Stated.	“Proposal to test rater quality” [1]	0%	0% 6% USED SITES EARLIER	0%	0%
47. Site Monitoring Described & Adequacy Evaluated *	“difficulties monitoring sites for conformity to protocols, undermine CTs'”[13, 14]. “ites falsifying data” [Authors]	0%	0%	10%	0%
48. Intra-study Rater Training provided*	[Authors] “observations in confidential reviews of CTs”	0%	0%	10%	0%
49. Advertising for Subjects Addressed	“Question of effects from advertising compared to self-selected patients” [1]	0%	0%	20%	0%
50. Sites Noted as Commercial or Not	“Question of possible effects from site priorities” [1]	0%	0%	0%	0%
51. Sites Noted as Academic or Not	“Question of possible effects from site priorities” [1]	0%	0%	0%	0%

Table II: 6. Protocol Controls of Methods and Clinical Transfer2,a,b
Methodological Query Is there discussion of…? Or relevant data reported about…?	Literature references	Approved AD Studies	Failed AD Studies	MCI Studies	AD Drugs in Development Studies
52. Are Critical Procedures Controlled by Protocols*	“latest and most rigorous methods for designing and executing study protocols” [6, p. ix]. ‘lack of adherence of raters and administrators to protocols’ [authors]	10%	6%	10%	0%
53. Are Protocols Referenced for SOM*	“Authorities urge caution using surrogate markers to infer clinical outcomes” [99-101]. “Operational protocols needed to overcome human and machine errors” [55, 61, 62]	0%	6%	0%	10%
54. Are Conditions for Effective Drug Use in Clinic Tested	“Knowledge transfer to clinic” [1, 102, 107]	0%	6%	0%	0%
55. Are Conditions for Effective Drug Use in Clinic Described for Practitioners	“Knowledge transfer to clinic” [1, 102, 107]	0%	6%	0%	0%
56. Are Clinical Decision Rules Developed and Tested	“clinical decision rules can provide effective guidance for health care policy making and management” [102-107]	0%	0%	0%	0%

¹As in other categories publications referenced may not be pivotal. For failed AD drugs no publication identified itself or could be identified by the authors as pivotal to the decision to abandon development.

²The primary justification for human research in drug development is inform patient care (Kralawish's 2006; Becker and Greig 2008; Kent and Hayward 2007; Califf 2005; Tunis et al. 2003; Rich 2002; Becker 2002); however, transfer of CT conclusions to the clinic seldom addresses the methodological limitations in CTs that also affect patient care (Becker and Greig 2008).

^aReferences [Authors] indicate that the subject of the query was developed by the authors without reference to specific publications.

^bThe % Compliant refers to the percent of drug developments compliant with the query aims. The % Compliant does not indicate the percentage of published reports compliant with the aims of the query since failures to publish in the plans for development, or about status or outcomes makes the drug development non-compliant with the aims or standards expressed in the query item.

^*Indicates conditions directly impacting on power.

Establishing dosing ranges, dose titrations, and effects on dosing from individual pharmacokinetic-dynamic diversity—difficult and crucial problems

Becker and Greig [1] and Pangalos [5] discuss recent CTs where patients were dosed below the dosing ranges shown effective in earlier Phase II dosing studies and confirmed in CTs. They emphasized the importance of establishing dosing parameters in early clinical studies, having evidence from animal studies that these parameters provide appropriate concentrations of drug at brain targets, and insuring that these dosing protocols control later CT dosing and drug management in clinical practice. Becker et al. [19, 20, 52] found dosing by weight necessary to approximate an identified optimal dose. Microdosing studies provide one effective way to confirm drug concentrations at brain targets otherwise open only to extrapolation from models developed in vitro and in animal studies [1]. Dosing studies must also take into account problems of titration and adherence that may occur in later clinical practice [43-46]. Bellelli et al. [47] found that 59% of community practitioners did not increase ChEIs from their lowest starting doses. A range of studies may be needed to adjust dosing for implementation under real world conditions and for the well known changes with aging that potentially affect drug concentrations at molecular targets. These changes include fat compartment increases, water compartment decreases, serum albumin decreases, α1-acid glycoprotein increases, altered hepatic P-450 enzyme system metabolism, decreased renal clearance, and so forth that impact pharmacokinetics, pharmacodynamics, as well as potential drug-drug interactions.

Drug kinetics and dynamics not adequately considered in drug developments

From what is published one can speculate that sponsors of drug developments in AD do not take advantage of traditional stepwise research preparing for CTs or use to full advantage pharmacokinetic and pharmacodynamic resources. It appears that in a rush to market Myriad, Neurochem, Sanofi, Elan, Bayer, and other pharmaceutical firms use post-hoc analysis, animal and not human dosing testing, biochemical marker changes, business priorities, and Phase III without preliminary evidence for dosing and efficacy at Phase II and earlier stages [1]. The authors found drug kinetics and dynamics critical to effective use of metrifonate and have argued elsewhere that disregard for basic pharmacokinetics and pharmacodynamics led to the failure of metrifonate to gain an NDA [1, 48-52].

In the literature we could not find extended discussions of the prevalence of or consequences from research sites falsifying data, lack of adherence of raters and administrators to protocols, investigators at sites lax applications of diagnostic criteria to secure subjects, the possible impacts of advertising for patients for a CT compared to patients who present spontaneously for diagnosis. We encountered each of these problems during participation in multicenter CTs, over the last 40 years confirmed their presence in informal discussions with colleagues and study monitors, and occasionally read of their importance [12]. The literature indirectly offers support for consideration of these issues, for example, Kobak et al. [25] find site raters exaggerating pathology apparently to qualify subjects for research studies, Carroll [53] views pharmaceutical firms ignoring subtypes of depressive diagnoses in CTs to gain unrestricted drug approvals, and Petersen and Morris [54] point out the increased heterogeneity in unselected epidemiological subject samples compared to those who seek help from a care provider for a problem, a difference that may also occur between self-selected patients and those persons responding to advertisements for research subjects.

Kelloff et al. [54] describe in cancer studies, and Pincus and Stein [10] in rheumatoid arthritis studies how inappropriate populations and settings prevent expressions of drug efficacy raising the risk of Type II interpretations of CT outcomes. While, with Raschetti et al. [55], these authors emphasize the importance of criteria to regularize practice across research sites, they do not call specifically for development of protocols to systematize, control, and replicate conditions and practices across centers or for investigators in centers [11]. Williams' [56] protocol for administration of the Hamilton Depression Scale allowed Engelhardt, Kobak, and their colleagues to uncover the inaccuracies introduced by outcome rating practices and in qualifying candidates as subjects for depression CTs [15, 25]. They took important steps towards operationalizing the concept of the careful interview Petersen and Morris [54] depend on for diagnostic validity. As we discuss below, failures to operationalize MCI criteria may, in part, account for a lack of success in those CTs [57].

Bias in test administration expressed in placebo controls

Imprecision in measurement can be expressed as the standard deviation of repeated test-retest administrations of a test under conditions that should not affect a subject's or measuring instrument's performance [21, 58]. Becker and Markwell [21] demonstrated that the standard error of measurement for test-retest data is a more sensitive indicator of potentially interfering error than are reliability coefficients. Inaccuracies in measurement require comparisons with evaluations produced by ‘gold standard’ raters or instruments. A further problem in our experience, but not for Ferris [59], is bias expressed in AD CTs by placebo group improvements on outcome measures over the period of the double-blind [3]. Becker and Greig [1] conclude that the demonstrated magnitudes of imprecision, inaccuracy, and bias found in AD and other CTs precludes long-term dependence of AD investigators on clinician rated outcome measures as endpoints for individual CTs. On the other hand, investigators such as Schneider [60], look to frequent use of improved sensitive neuropsychological tests to overcome difficulties with current CT practices. Becker and Markwell [21] demonstrated that using the mean of ratings, rather than individual ratings, improves precision and power to detect statistically significant differences between treatment arms [3]. Kobak et al. [37] and Targum [38] demonstrated improved accuracy with extended intensified rater training. The effectiveness of each of these methods depends on the manner in which they are used. In studies using neuropsychological tests, biochemical assays, imaging and, even sporadically, in applications of diagnostic criteria and uses of ratings, investigators specify with protocols the conditions for use of the test [11, 54, 56, 61, 62]. Generally, extensive protocols to control practices are not provided in CT proposals or reports.

Overview of problems in MCI drug development

Petersen [57] observed how study design vulnerabilities due to heterogeneity of the sample, insensitivity of outcome criteria for progression, subtherapeutic dosing, and long trial length are able to account for the trial's failure. Jelic [109] cautioned that heterogeneous samples at entry to an MCI CT, lack of biomarkers to enrich samples with subjects at risk of deterioration, suboptimal durations of treatment, and ineffective indicators of disease progression must be overcome in future MCI CTs. Harrison et al. [31] developed the Neuropsychological Test Battery to overcome the ADAS-Cog's insensitivity to change in MCI. This test is not practical for use in other than specialized clinics. With the evidence of neuropathological changes probably irreversible prior to appearance of adequate symptomatology or brain volume changes to differentiate pre-AD from normal aging, AD research may have to shift from clinical ratings to practical screening biomarkers to best prevent disabilities from AD. Diagnosis that depends on tissue loss, such as hippocampal atrophy, offers only secondary, not primary prevention and accepts, does not anticipate and prevent, irreversible changes [68-72]. Given the neuropathological change prior to rated cognitive changes and the imprecision of the ADAS-Cog and MMSE, practitioners' clinical judgments, with or without support of testing, are probably not adequately reliable to ground either AD or MCI patient management decisions [14]. A truly most effective preventive intervention for AD will most probably require patient management based on results from biomarkers sensitive to pre-clinical changes predictive of later AD.

Setting drug development priorities

In compiling a list of AD and MCI drug candidates, Becker and Greig [1] found that most AD drugs, with the exception of some traditional remedies such as gingko bilboa, were studied with reference to a specific mechanism of action, for example acetylcholinesterase inhibition, Beta sheet breaking, gamma secretase inhibition, and so forth. Even though AD drug candidates may be active through over 30 different mechanisms, only ChEIs and memantine, an NDMA receptor glutamate antagonist, received US Government Food and Drug Administration (FDA) new drug approval. For example, after adding to the list used for Becker and Greig [1] we easily identified 14 ChEIs that reached clinical testing in AD with 5 approved, 2 under development currently, and the remainder abandoned during development. Becker and Greig [1] discuss the apparent errors in development that appear to have caused the failures of two drugs while others, such as heptylphysostigmine, failed for toxicity of the molecule separate from the mechanism of inhibition. We identified 3 glutamate antagonists with 1 approved and 3 abandoned; 15 muscarinic agonists with none approved and all abandoned, and 12 nootropics, 9 anti-inflammatory drugs, and 4 choline precursors failed in CTs or earlier in development.

It seems generally accepted that ChEIs, as a class, share a mechanism of action potentially beneficial in AD in the absence of toxicity due to the molecule [51] while, with the possible exception of glutamate antagonists, the mechanism(s) of action for each of the other classes, muscarinic agonists, nootropics, anti-inflammatories, precursors of choline in studies to date demonstrate no functional benefits in AD. Given the limited professional and financial resources available to support clinical AD research and the ethical problems of exposing patients to compounds not well justified for further research, we would expect CTs testing compounds, thought to act with previously failed mechanisms, to be provided specific justifications for why this compound should be regarded as a possible exception.

Subject samples

Raschetti et al. [55] reviewed AChEI CTs in MCI. They found samples being not equivalent across studies impairing cross-study comparisons and potentially contributing to failed outcomes. Given Kobak et al.'s [25] evidence that evaluators rated depressed subjects at baseline more severely than outside observers blind to the situation, it is possible, for example, that Feldman et al. [108] recruited less severely challenged MCI subjects from investigators being under pressure to fill quotas and from problems with entry criteria. Clinical skills affect evaluation abilities [33, 63]. Study sites and investigators need training and evaluation in use of entry criteria, including AD diagnostic criteria. Visser et al. [115] found different samples drawn from the same population in their clinic after applying entry criteria used for different MCI studies. Varma et al. [26] demonstrated that the homogeneity of AD research subject samples is inevitably compromised by the inability of NINCDS-ADRDA criteria to exclude frontotemporal dementia cases.

Biomarker research

Currently magnetic resonance imaging (MRI), positron emission tomography (PET), single positron emission computerized tomography (SPECT), and spinal fluid biochemistry offer promise as potential sources of quantitative biomarkers for AD disease progression. PET, SPECT, 1H MRS, and MR volumetry of the hippocampus can distinguish AD patients from elderly normals [68]. Hippocampal volumes in MCI have been shown to predict progression to AD [69-74]. Longitudinal trends in brain images have been associated with cognitive decline [68]. Nordberg [75] interprets these preliminary successes as showing promise for imaging to support diagnosis and indicate disease severity in AD. Acceleration along this path can be expected from the multi-center coordination of development with the Alzheimer's Disease Neuroimaging Initiative (ADNI) [62]. Currently the methods of choice for imaging AD pathologies have not been identified [76].

Hippocampal atrophy, more prevalent in patients disposed to later AD and in AD patients than in normal aging, correlates with disease severity, but not adequately to show drug effects on disease progression in periods of less than one year [77-79]. Longitudinal cortical grey matter atrophy in AD witnesses to the need for early diagnosis and treatment [78, 80-82]. PET, able to indicate reduced regional cerebral metabolism in AD with fluorodeoxyglucose (FDG) imaging, is useful in AD diagnosis and can assist in excluding frontotemporal dementia [83-87]. Pittsburgh compound B (PiB) and FDDNP allow visualization of amyloid [83, 86]. Stable PiB retention after 2 years supports early amyloid accumulations in AD possibly prior to declines in glucose metabolism and cognition [88]. Unfortunately no imaging techniques yet predict for an individual case [89].

In AD amyloid β (Aβ) CSF levels decrease, tau fractions increase and BACE1 levels in MCI predict increased risk for developing AD [72, 90]. Phosphorilated tau correlates with neurofibrillary tangles (NFT) and with hyperphosphorylated tau concentrations in post mortem AD brains [91]. Unlike Aβ concentrations that remain stable, t-tau increases from early to advanced AD [92-94]. Hansson et al. [95] proposed the Aβ 42/Aβ 40 ratio as a predictive biomarker for AD [96, 97]. Given the rapid progress recently applying imaging and molecular biomarkers to our understanding of AD, it seems reasonable to evaluate whether or not the move to biomarkers can, in the near future, overcome current impediments to most effective uses of CTs in AD drug development. The FDA anticipates that biomarkers will become increasingly important in all phases of drug development [98]. Authorities urge caution using markers to infer clinical outcomes [99-101]. Since the FDA Modernization Act of 1997 allows for fast track drug approval when a surrogate marker indicates a drug as most likely to provide clinical benefit for serious and life-threatening diseases, markers may come to play an important role introducing new AD drugs [67]. As Temple [64] emphasizes, a validated surrogate marker must demonstrate long-term safety in addition to confirmations of predicted efficacy.

As we already noted, hippocampal volumes in MCI have been shown to predict progression to AD. BACE1 levels in MCI predict increase risk for developing AD [90]. Brys et al. [110] found P-tau was the strongest among the CSF biomarkers they investigated for predicting decline from MCI to AD. Combined measurements may show promise for quantifying drug effects on pathological processes in AD and pre-AD [111-113]. Given the failures of drugs in MCI, MCI both should not be treated as a clinical entity and can not at this time be treated creating a barrier to early interventions in AD [114]. On the other hand, research suggests that investigators will develop biomarkers with greater predictive capacities for quantifying progression, targets for treatment that hopefully, as surrogates, will overcome current unreliabilities and the complications for CTs and patient care.

Proposed modifications to methodologies: developing and using new CT technologies

Based on the range of difficulties designing, executing, analyzing, and interpreting AD CTs, Becker and Greig [1] proposed that investigators may, in choosing how they develop AD drugs, fail their drugs by not providing fair tests for efficacy and adequate safeguards against compromises to safety [3, 57]. They concluded that limitations inherent in AD drug candidates do not necessarily account for all failures of CTs to confirm efficacy [1, 3]. Because of dependence on rating scales, the risk of Type II error may be more prominent in AD drug development and CTs and in other areas of neurology and psychiatry where investigational outcomes are vulnerable to human errors. In accord with Pangalos et al. [5], we suspect methodological factors account significantly for the low success rate of AD drug developments.

In response to the difficulties they and others uncovered, Pangalos et al [5] and Becker and Greig [1] proposed modifications in drug development and CT practices designed to reduce methods and practices interfering with demonstrations of efficacy or other outcome aims. Evidence indicates that some methodologies and practices may be flawed in AD drug development. Currently the evidence is only circumstantial and more research is required into specific problems: random and systematic errors; bias; the limitations on sensitivity and specificity of rating scales; and limitations on human abilities to use rating instruments with accuracy and without bias; and other factors. It seems unlikely that research can overcome the problems that arise from exclusive dependence in CTs on rated outcomes. Alzheimer Disease Neuroimaging Initiative investigators are currently researching imaging and biochemical biomarkers, their use as potential surrogate outcomes, and protocols for effective applications in research and patient care [11, 61, 62]. Similar concerns for the reliability of clinical assessments arise in clinical practice where, without rigorous training in evaluation, often with limited skills and experience with AD, and commonly without using established rating instruments, practitioners reach diagnostic and management decisions. Accuracy in diagnosis and management assessments varies with the experience of the rater [33, 63]. Problems with diagnostic and rating scale evaluations will not be automatically corrected by advances in rating and testing methods in CTs. For example, the Neuropsychological Test Battery, required to overcome sensitivity barriers in MCI research, where disease modifying interventions will become available for practitioners if research is successful, is impractical in the clinic because of time and skills needed to perform the tests [31].

As a resolution to the difficulties they found inherent in AD CT methods and practices, Becker and Greig [1] proposed intensive development of quantitative biomarkers able to provide molecular outcome targets for AD CTs. They reasoned that, although the FDA appropriately requires demonstrated clinical efficacy in two areas for use of a CT in support of a new drug application (NDA), limiting outcomes in this way, the risks of Type II errors from AD clinician rated outcomes are too great. Using quantitative and pathology-specific biomarkers, such as markers for amyloid, tau, cell death, and so forth, Becker and Greig anticipated the possibility of first confirming molecular effects from an AD drug and then exploring the molecular effects or generalizing from other evidence of clinical benefits associated with similar molecular changes to establish the relationship to clinical benefits. Separate trials, focused on clinical benefits associated with specific modifications of pathology, could then be used to confirm that the CTs with surrogate markers comply with FDA NDA required clinical benefits. Under this proposal a drug may become a useful pharmacological probe or indicator of a class action even though it does not itself offer clinical benefits.

Becker and Greig [1] proposed for consideration three new procedures meant to overcome or reduce risks of Type II errors in interpretation of AD CT results, to reduce interference of real world conditions with determinations of efficacy of a candidate AD drug, and to use a CT as a model defining the conditions of clinical use of the drug required to maximize the opportunity for the drug to be effective with a practitioner's patient. The proposed new procedures involve modifications to currently preferred intention-to-treat (ITT) based analyses of CT data meant to increase the effectiveness of CTs testing for drug efficacy, wider uses of protocols to govern CTs and clinical practices to assure scientific accuracy, precision, control, systematization and reliability, and rapid development of biomarkers as quantitative surrogate markers [9, 64-67]. Raschetti et al. [33] and the Alzheimer's Disease Neuroimaging Initiative investigators, Foster [61], Thal [11], and Weiner [62], find protocols necessary to control problems of human and machine errors intrusions into AD research.

Objectives

Since 1997 the authors have studied how better scientifically to systematize and control AD CTs and patient care. Consistent with Kralawish's recent NEJM editorial [102], the authors emphasized the role of CTs informing patient care and investigated methods to strengthen that information transfer function of AD CTs [1, 107]. Given the methodological and practical issues that potentially interfere with AD drug developments providing reliable tests of drug efficacy and that open risks of Type II errors in interpretation of research outcomes, for this paper we sought to first, confirm the presence of methodological and practical problems in a selection of studies taken from AD drug developments and, second, to bring together our own and other's recommendations for changes able to overcome these problems and the interference of problems with AD CTs applying to individual patients in clinical practice and informing health care policy [103-106]. This paper aims to estimate from a small random sample of the literature, the presence or absence of possibly critically important methodological deficiencies able to interfere with AD drug development successes and to justify corrections to methods and practices able to overcome this interference.

Methods

For this study we organized our assessments of drug development decision making, methods, and practices under six broad activities in drug development: 1) mechanisms of action, pharmacological activities, design of studies and publication; 2) dosing; 3) research subject samples; 4) outcome measures; 5) research sites and investigators; and 6) protocol controls over methods and practices and clinical information transfer.

These topics were chosen from issues raised in the Background and used to guide our analysis of selected AD drug developments. One major problem, encountered by anyone wishing to evaluate why some AD drugs succeed and others fail, is that most negative outcomes of drug developments are not published and sponsors refuse to release information about the development [1, 5, 16-18, 55]. Consequently, in earlier publications [1, 3, 14] we retreated to analyses of selected reports, personal experiences with drug developments, and other available information that seemed relevant. In this study, using random sampling, we planned to estimate more precisely if and how extensively not publishing investigational outcomes interferes with advancing AD research and patient care.

To update our list of AD drug developments used in Becker and Greig [1] we searched Medline, the Cochrane Collaboration files, the Cochrane Collaboration Trial Register, the Food and Drug Administration controlled clinical trials registry, Current Controlled Trials, clinicalstudyresults.org, clinical trial result postings by sponsoring drug companies on their websites, references in publications, review articles, and investment banking firms' and other published lists [116] of AD drugs considered for investigation or commercial development. We then used a random number table to select drugs and specific studies of these drugs to analyze. After a drug was randomly chosen for inclusion as one of the 10 drugs to be addressed in a category, we randomly chose, whenever possible, among pivotal studies or later reported studies of the drug. When a pivotal study report or study publication from the last reported or, for drugs in development, current phase of development, was not available, we first chose a registry report of results from the appropriate CT, second the registration as a CT, and then third the most recent published investigation of the compound - randomly chosen from a group when more than one publication was available. As a result we realized we would underestimate the rate of failure to publish negative trials in the interests of sampling a range of studies in each of four categories. We developed four separate randomly selected lists, each with 10 papers providing information on drugs: 1) AD drugs approved and used in clinical practice; 2) AD drugs reported failed in drug development or presumed abandoned because of no published references to outcomes from earlier reported research intentions; 3) MCI drugs failed in drug development, and 4) drugs currently under development for AD or MCI.

From our survey of the literature and extrapolations to clinical research from relevant good laboratory experimental practices we then developed a list of 56 specific queries covering the six issues where our survey of the Background literature opened possible methodological impediments to AD drug development. Each of the 56 queries was worded to investigate a specific issue raised in our survey of background literature (See Tables II:1-6). One of the authors rated each selected study for whether each of the 56 topics is reported in the publication selected for inclusion in this study and for how the report in the publication addresses the subject matter of the topic. The other authors reviewed the ratings.

Studies were judged using each query based on whether or not the publication discussed, or at least acknowledged, the specific subject of the query. This produced for each category the percentage of publications in which the subject area was addressed or at least noted. Where data could be compiled, such as numbers of subjects (Table II: 5. Query 39) or numbers of sites (Table II: 5. Query 41), this data was recorded and summarized for each of the four categories of study outcomes.

Results

Publication of drug development outcomes

We randomly chose, when available, 10 publications from each of our four categories of drug outcomes. We had to survey the literature for 18 compounds to obtain 10 published reports or acknowledgements of outcomes for unapproved compounds previously reported as being developed for AD. The drugs and references reviewed are listed in Table I. After the random selection was completed we noted Aisen et al. [148] was mistakenly assigned under MCI trials. We retained the trial because the study addresses possible differences between NSAIDs reducing risk of subsequent AD in MCI and stabilizing AD which we saw as relevant to study subsequently of anti-inflammatory effects on MCI syndrome mechanisms of progression.

Table I.

AD Drugs Randomly Selected for Review and, When Available, Publications

Approved Drugs	Drugs Failed or Abandoned in Development	Studies of MCI	Drugs Currently in Active Development
Tacrine [126-128] Donepezil [129-131] Revastigmine [132] Memantine [121, 133, 134]	Ipidacrine, Sabeluzole [135], ABT-418 [136], Milameline [137], Alvameline [138], Zifrosilone [139], Vitamin E [117], CX 516 [140], Oxiracetam [141], Propentofylline [118], SC-110, Alfoscerate [120], XE991, Diethylspermine, EGb 761 [119], Neotrofin, Tesofensine, PPI 1019.	Donepezil [142, 143] Rivastigmine [108] Galantamine [144, 145] Celecoxib [146] Piracetam [147] Rofecoxib [148] Estrogen & Progesterone [149] Aspirin & vitamin E [124]	Huperzine A [150] ACC-001 [151], TTP 488 [152], MCP-7869 [153], MK 0752, Cerebrolysin [154], PBT2 [155], Vitamins E & C, alpha-lipoic acid, & coenzyme Q [156], Docosahexaenoic acid [157], Tramiprosate (Alzhemed) [158].

Legend for Table I: Drug and [references to specific publications included in this study]. When no references are provided this indicates the drug was randomly chosen as an identified AD development candidate but lacks publications reporting outcomes from the intended research.

Our selection gives us an estimate of 56% of compounds, claimed to be under development for AD, without any information available about outcome. We found publications of results at the level of the last reported stage of development for only four compounds of the 18 failed AD compounds, Vitamin E [117], propentophylline [118], ginkgo biloba [119], and alfoscerate [120]. None of the reports identified itself as a pivotal study for the stage (Table I). This provides us an estimate of 22% of compounds in our AD development sample as having some data available from the last reported stage of development and possibly no data about specific studies that led to abandonment of the drug (Table II: 1. Query 11).

For AD approved drugs we found 9 or 90% of randomly selected CTs, published. For MCI 70% of our randomly selected trials were published and for drugs currently under development, except for registrations of CTs, no publications about the current stage, as could be expected since the latest stage of development remains in progress. Assuming that unpublished failed AD drug investigations are negative and, when not referenced in the literature that they underlie abandonment of these drugs, we have only 7 negative pivotal publications in MCI, an estimated 24% publication rate for negative outcomes in AD drug development—1 negative study for an AD approved drug [121] 18 failed AD drugs and 10 failed MCI applications.

Attention of investigators to potential areas of methodological interferences with development

In the total of 40 drugs with some literature on outcomes of development we found 0% descriptions of methods and procedures in each of the four categories of our sample for 14 out of our 56 queries of trial reports (Tables II:1-6), a maximum of 20% to >/=0% reporting in 22 of 56 queries, and a maximum of 50% but greater than the range 20% to >/=0% for 2 queries. This estimates that less than a third of the areas of methodological concern we raise ever receive attention in more than 50% of AD investigations. In fact, higher compliance is often reached in only one out of the four outcome categories, suggesting much lower overall levels of investigator concern with the potentially problematic methodological issues that we and others have identified.

Establishing and adhering to a well-evidenced effective dosing range

For the 10 queries we developed to reflect commitments to sound dosing development and practices (Table II: 2), only stating the basis for dosing increases in a CT (Table II: 2. Query 17) and reports of the percentage of subjects reaching target doses (Table II: 2. Query 18) ever exceeded 50% compliance. Six queries (Table II: 2. Queries 12-16, 19) were addressed in 20% or less of the reports we reviewed.

Characteristics of research subjects

Measures to insure sample homogeneity within a study and comparability of study samples with samples drawn for other studies were reported in less than 50% of failed AD and AD in development descriptions (Table II: 3). Although we found criteria listed for each of the 10 MCI studies, the variations among criteria sets, lack of operational definitions [55], and the non-equivalent samples drawn by Visser et al. [115] using the different criteria sets demonstrates how our requirement of attention or description does not tap into the full vulnerabilities of CTs to methodological deficiencies. Consideration of effects on outcomes from other diagnoses and medications were described in 20% or less of the papers and imaging, although required in 20-30% of studies, specifically addressed confounding of the sample by fronto-temportal dementia (Table II: 3. Query 26) and confirmation of AD (Table II: 3. Query 27) in 0% of reports.

Attention to rater reliability and effects from unreliability on sample size, numbers of sites, and so forth

Except for reporting the outcome measures used (Table II: 4. Query 28), none of the other 8 queries (Table II: 4. Queries 29-36) addressing accuracy, precision, or bias as sources of error exceeded 20% consideration in study reports.

One hundred % of papers reported the number of subjects in the study (Table II: 4. Query 39). Information about sample sizes is compromised by the unpublished failed AD studies. Twenty to 70% of papers identified subjects lost during the study (Table II: 4. Query 40) and, leaving out unreported failed AD studies, only 60 to 90% of publications reported the number of involved sites (Table II: 4. Query 41). No reports provided information on the number of sites providing less than 6 subjects to the study (Table II: 4. Query 42). Criteria for qualification and disqualification of sites (Table II: 4. Queries 43-46) were reported by 0% and monitoring activities (Table II: 4. Query 47) addressed in only one paper.

Research sites and investigators

Rater qualifications and site monitoring are reported in Table II: 5.

Protocols to control methods, practices, and conditions

Zero to 10 % of papers described protocols to control the manner in which tests, rating scales, criteria for diagnosis, recruitment, biomarkers or other features of the investigation are applied, used, and interpreted (Table II: 6. Queries 52-53). Studies reporting protocols almost exclusively addressed methods for neuropsychological testing or biomarker assays.

Information transfer to practitioners

Zero % of studies described methods to transfer methodologies, practices, protocols, and procedures (Table II: 6. Queries 54-56) from the CT to the practitioner.

Discussion

Science presumes free sharing of information as essential to orderly progress in its knowledge, an ideal more easily realized with current access to information technologies [Young 122]. Our sample of information on outcomes of 48 compounds identified as under consideration as potential AD therapeutics supports earlier claims that drug developments with negative outcomes go unreported. This lack of information about failed drug developments blocks assessments such as we have undertaken to evaluate how methodologies might contribute to drug failures and similar inquires into the pharmacology and therapeutics of AD drugs. As we have proposed elsewhere [1] and as Kralawish [102], Chalmers [18], and others have protested for decades, fair evaluations of the literature on AD drug therapeutics and clinical pharmacological methods are blocked by the biases introduced by the selective reporting of results from scientific pharmacological investigations in AD. Clinical investigators often must work in the dark as commercial sponsors withhold basic pharmacological information about a drug under investigation as proprietary [12]. We suggest that our data add to the body of reporting that calls for core revisions in AD researchers' and practitioners' access to results from investigations. The establishment of clinical trial registries meant to improve reporting from negative outcome investigations is not succeeding according to our data.

Science presumes conditions of investigation that are controlled and described such that other scientists will be able to replicate the original investigation. In our survey of the literature on potential methodological compromises in AD CTs, we find, in the failures to report methods to control possible interference from sampling of subjects, protocols for applications of criteria for admission to a study, consideration of unreliability due to problems with precision, accuracy, sensitivity, selectivity, investigator bias, and other factors. Neuropsychological, imaging, and biochemical studies alone in our sample provided the detailed descriptions of methodologies and conditions for applications of methodologies that allow readers to evaluate the control of conditions in an investigation and provide information needed to replicate the investigation. In our sample of 48 candidate AD compounds we found very low rates of reported attention to the majority of factors highlighted for us from our survey of the literature on methodological risks to validity and our own research and experiences. Within discussions of studies that failed or proposals for investigations we found few, if any, considerations in depth of how methodological flaws in the trial could lead to Type II errors.

Becker and Greig [1] and Becker [3] questioned whether or not drug development and CTs failed the drugs they tested because methodological deficiencies increased the probability of Type II errors. The low rates of attention we found to methodological issues that could invalidate drug development investigations, such as unreliability that leads to variance, reduced power, large numbers of subjects to meet power requirements, large numbers of sites to provide subjects, heterogeneous samples, inadequate monitoring and re-training of site personnel during studies, and so forth, sustain the concern that current AD CT methods and practices may lead to rejection of compounds that could be efficacious in AD or indicative of mechanisms of drug action efficacious in AD. Based on this added evidence we call for further research into AD clinical pharmacological methods and practices with the aims to make AD drug developments more effective, to increase confidence warranted in reported outcomes, to provide researchers improved tools for use in areas of special concern, such as outcome measure reliabilities and controlling sample sizes, and to transfer to practitioners the methods, procedures, practices, protocols and other conditions of use of a drug shown in CTs as conditions for drug effectiveness. Readers need increased confidence in the validity of CT reports; however, to meet the clinical practice aims of medical research, improvements in health care, CTs need successful transfer to clinicians of information on how to use drugs effectively in patient care.

Becker and Greig [1] proposed, based on the available literature, that human error and limitations inherent in rating scales and neuropsychological tests precluded these tools from ever providing adequately precise and accurate information for CT testing of drugs and patient management with approved drugs. They suggested specific tightening up of current CT practices, further research into development of protocols to govern applications of methods and practices in research and the clinic, modification of intention-to-treat designs to allow CT investigators to change real world conditions that undermine demonstrations of effectiveness of AD candidate drugs and effective uses of approved drugs with patients, and accelerated development of quantitative biomarkers able to be used as surrogate outcomes in CTs and clinical practices. Imaging and biochemical monitoring currently offer specific advantages to AD researchers and practitioners, for example, exclusions of non-AD dementias and confirmation of AD [11]. These biomarkers can not currently serve as surrogate outcomes; yet, the increase we find in use of biomarkers as secondary outcome measures with drugs under development, 60% of studies sampled (Table II: 4. Query 37), and Phase II biomarker-based justifications for proceeding to Phase III [123] are in most proposals not accompanied by considerations of the validity of the proposed use (Table II: 4. Query 38). Advances proposed in methodologies, such as the use of the NTP battery for increased sensitivity to cognitive impairments, are not practical for clinicians who must depend on cognitive, behavioral, and global assessments to manage dosing and select among drug alternatives available to them when patients appear not to benefit from a current therapy.

As we earlier proposed, real improvements to CT validity and effective investigational information transfer to clinicians may await qualification of AD biomarkers as surrogate outcomes and practical guidance and community resources to support their use in AD and pre-AD patient care. One barrier to increased orientation of CTs towards biomarkers as outcome variables is a possible conflation of concepts of disease and patients' clinical status. A drug acts at molecular targets. Molecular targeting recommends biomarkers of molecular pathologies directly linked to clinical illness as preferred measures of outcome in CTs and for management in the clinic. Clinical benefits to patients and safety remain crucial as goals of treatment; however, as with many chronic diseases, prevention of early progression to AD and its disabilities may depend solely on management of biomarkers shown in long-term studies to be linked to later clinical illness. Given the evidence of pre-clinical progression of pathologies such as abnormalities in amyloid metabolism, we expect that early AD interventions will be managed based on biomarkers and not clinical assessments. Clinical assessments will test biomarkers for efficacy predicting long-term outcomes from molecular disease; drugs, through activities on systems monitored by biomarkers, will prevent clinical illness, disability, and handicap.

Our search may have missed publications and we thereby overestimate rates of failed publications. Our queries and methods are not tested and many aspects could be open to appropriate criticisms and refinements. On the other hand, we doubt that the full magnitude of inattention we found can be questioned and find support from our study for the results in already published reports that we used to develop our queries (see Tables II: 1-6). We find nothing in our current study that would undermine already existing concerns that methodological deficiencies and inadequately controlled procedural practices may be interfering with effective AD drug development and effective uses of AD drugs with patients in the clinic. Given the low rates of success with AD drug candidates we repeat our earlier proposals that priority be given to research into AD clinical pharmacological methods, qualification of biomarkers as quantitative surrogate outcome measures, and redefinitions of CTs as providing both evidence of efficacy and of the conditions of use required in clinical practice to achieve efficacy.

Becker and Greig [1] questioned whether symptoms and adverse events associated with other diseases and medications could be mistaken by investigators as AD drug associated and account for failures to raise doses of AD drug to within prescribed ranges. We find high rates of other diagnoses being present and of non-AD medications prescribed in the studies we reviewed (Table II: 3. Queries 23 and 24). We do not find a clear association of these factors with failures to achieve target dosing (Table II: 2. Query 18) or with excessively high reported rates of discontinuation from drug due to adverse events (Table II: 2. Query 20). On the other hand, a proper investigation of this relationship is impossible due to the sporadic reporting in these areas of query.

The IOM report on PTSD research [6] claimed that many of their criticisms could be accounted for by earlier PTSD studies not providing the quality CTs currently available. If one assumes that roughly for our categories the AD approved drugs tap earlier AD research practices, AD failed the next period, MCI the early 21^st century, and drugs under development current practices we see no trends on the majority of queries that would suggest increasing methodological soundness over time. We take this as suggestive that the IOM claim of improving research practices with time may not apply for AD and, given the references to failed methodologies in depression studies we reviewed as background, may be questioned for PTSD without direct evidence for that diagnosis to the contrary.

Specific attention may be required to the use of reliability coefficients to justify confidence in assessment methods. For example, Kang et al. [124] accepted test-retest correlations of 0.7 and 0.81 as validations of their interview methods for estimating cognitive status; yet, Becker and Markwell [21] found test-retest reliability coefficients ranging from 0.85 to 0.95 associated with levels of random measurement error greater than the average expected drug effects or disease decline over one year. Outcome measures must meet the levels required by the study for specific aspects of reliability, that is, precision, accuracy, sensitivity, specificity and so forth. The validity of indicators of reliability is too often neglected as part of research studies or proposals (Table II: 4. Query 33). We interpret the evidence from Becker and Markwell [21] for hidden random measurement error effects in data and from Engelhardt et al. [15] and others for widespread rater inaccuracies as requiring more than intra- and inter-rater reliabilities to ground validity for an investigation.

Our impression from our own work and the reports of others is that, although major steps away from current concerns with methodological reliability and validity await further research, much could be done immediately by greater attention to clinical pharmacological methods and practices. There is evidence that in both commercial developments and academic studies, as exemplified in recent MCI studies, before rushing to Phase III little or no attention is given to traditional Phase II aims to identify an effective dose and dosing range for a drug. For example, in the Feldman et al. [108] study of rivastigmine for MCI, it is not possible to decide whether the maximum 6 mg. dose achieved provides a fair test of the ChEI in MCI or under-dosing of subjects. As we earlier proposed for ChEIs [125] and as we demonstrated for metrifonate [19], ChEI may exhibit an inverted U shaped dose-response curve both within a population and within the range from normal to AD populations. Since there is most probably less cholinergic deficit in pre-AD than in AD, doses of ChEIs lower than those needed in AD may be indicated for MCI. Without Phase II preparations for the MCI studies the dosing issue can not be settled based on pharmacological evidence leaving unanswered questions of possible Type II error interpretations due to mistaken dosing in current ChEI MCI studies.

Although studies of unreliability in outcomes of CTs dependent on rating scales are sparse we interpret a trend in investigations that supports very difficult, if not impossible, problems to overcome. Evidence of imprecision, inaccuracy, bias, lack of skills in raters especially in large studies, accrues to convince us to take seriously that human errors and probably limitations in rating scales, in neuropsychological tests, and in the sensitivity of clinical symptoms to neuropathological losses in pre-AD, preclude AD research from continuing its dependence on rated patient outcomes. In our view, current evidence supports that, as a disease, AD first expresses itself in pre-symptomatic neuropathological losses that are irreversible. This justifies the current emphasis on early diagnosis and early treatment but also undermines our dependence on rated symptomatology because research and interventions in CTs and the clinic will want to occur while persons are still asymptomatic or so slightly impaired as to be indistinguishable from many normal aged persons. Consequently we call for research into biomarkers and their developments as surrogate outcome to be given highest priority.

Many of the problems plaguing AD research currently—large sample sizes, multiple sites, sites without raters skilled in AD evaluation, problems of monitoring sites, low power and consequent risks of Type II errors—we find secondary to human error intruding into AD assessments. To reduce these error interferences we suggest a two phase drug clinical development model. Biomarkers, as surrogate markers of neuropathological changes leading to AD and its clinical progression, will become the outcome measures for CTs. Thal et al. [11], when they demonstrate the greatly reduced numbers of subjects needed to achieve required CT power, witness to the improvements in accuracy and precision achievable with carefully conducted biomarker studies. In a separate phase of research we propose investigators qualify biomarkers as surrogate markers for patient benefits. These latter activities can be carried out in AD specialized centers with well-experienced staff trained in AD assessment and evaluation and practiced regularly in these skills. With a range of the measures we have identified—ongoing training, qualification of raters, use only of highly skilled and experienced raters, use of means of three data points rather than single data points, long-term outcomes, such as onset of AD and so forth—we anticipate small but effective studies linking biomarker status to clinical benefits for patients [1]. Larger studies with community sites can develop and test protocols able to provide the reliabilities for biomarkers needed for efficient CTs and practitioners' uses with patients. Smaller studies qualify these biomarkers as surrogates and thus as acceptable outcomes for regulatory authorities.

We interpret out current study of a sample of AD drug outcome reports as adding to Aisen's, Vellas', others, and our own concerns with too many centers, too big variations among centers, difficulties finding homogeneous groups and managing large numbers of subjects in a CT, and other potential interferences with AD drug developments. We also find additional support for the proposal by Becker [3] and Becker and Greig [1] that problems of variance and its consequences can not be effectively overcome so long as CTs remain dependent on human rated outcome measures. We conclude that the summary by Thal et al. [11] of estimated sample sizes needed when biomarkers are used, combined with our and others evidence supporting how human error intrudes into effective AD drug development and evidence of those intrusions in this report, supports giving high priority to research into establishing biomarkers as quantitative and reliable indicators of long-term efficacy in AD CTs.