Systematic reviews and lifelong diseases

Systematic reviews of randomised controlled trials provide an evidence base for treatment but too often fail to give adequate information on long term outcomes. Elphick and colleagues discuss the limitations of the systematic review of randomised controlled trials for patients with chronic or lifelong diseases and suggest that long term observational studies have a place in the evaluation of the benefits and risks of treatment

Synthesis of evidence from systematic reviews of randomised controlled trials is considered the gold standard when evaluating the effectiveness of treatments, yet systematic reviews do not always place as much emphasis on information about adverse effects or safety issues.¹,² In people with chronic, or lifelong, diseases long term outcomes are particularly important but are much less likely to be evaluated in randomised controlled trials. We discuss the results of a recent systematic review of randomised controlled trials of antibiotic therapy in cystic fibrosis, which provided preliminary information about antibiotic resistance but not about other adverse effects. Although the review suggested that this finding should be investigated by a further clinical trial, we discuss why this may be neither the most feasible nor the most efficient study design with which to evaluate long term outcomes in lifelong diseases.

Summary points

• A systematic review suggested that there was an increase in antibiotic resistance in patients treated with one compared with two antibiotics

• Systematic reviews of randomised controlled trials are often unable to provide adequate information about long term outcomes, which are important for people with chronic diseases

• A tension exists in randomised controlled trials between evaluating short term outcomes, which may indicate a need to change practice, and continuing the trial for sufficient time to evaluate long term outcomes

• Databases specialising in a particular disease can play a part in capturing information on patients prospectively, provided the clinical questions are established from the outset

• To adequately evaluate benefits and risks of treatment for chronic diseases, systematic reviews should consider data from observational studies as well as from randomised controlled trials

Methods

We carried out a systematic review to assess the effectiveness of single compared with combination intravenous antibiotic therapy to treat respiratory exacerbations in people with cystic fibrosis.³ Here we focus on the comparison of a β lactam antibiotic with a β lactam and aminoglycoside combination. To investigate further possible long term adverse effects of these antibiotics we undertook a search of Medline from 1966 to identify studies that investigated toxicity related to aminoglycosides in patients with cystic fibrosis.

We included nine randomised controlled trials (386 patients) eight of which compared a β lactam antibiotic with a β lactam and aminoglycoside. Overall the quality of the methods was poor, and meta-analysis was not possible for most of the outcome measures.⁴ We found no significant differences between the therapies for lung function, symptom scores, and adverse effects. Three trials assessed the development of resistant strains of Pseudomonas aeruginosa.^5–7 We found no difference between the groups at baseline and immediately after treatment. Two to eight weeks after treatment, single therapy was associated with an increased number of patients whose sputum grew resistant strains (odds ratio 2.54, 95% confidence interval 1.13 to 5.68). The change in number of patients with resistant strains from baseline to follow up also showed a difference favouring combination treatment (4.07, 1.33 to 12.43; figure).

Use of antibiotics

Intravenous antibiotics are widely used to treat respiratory exacerbations in patients with cystic fibrosis, particularly when associated with P aeruginosa infection. The type, duration, and number of antibiotics used varies. A survey conducted in the United Kingdom and Ireland in 1993 found that 5 of 26 clinicians dealing with cystic fibrosis used a β lactam antibiotic alone as intravenous therapy for acute respiratory exacerbations due to P aeruginosa.⁸ The proportion of centres using monotherapy is likely to have decreased since then, as guidelines from the UK Cystic Fibrosis Trust recommended that two intravenous antibiotics should be used for a minimum of 10 days.⁹ This recommendation was made before the publication of our systematic review. The evidence to support it came from a study showing that a high proportion of children in one cystic fibrosis clinic had an identical strain of P aeruginosa, presumably transmitted by cross infection.¹⁰ The authors speculated that the policy of using ceftazidime monotherapy may have contributed to the emergence of this resistant organism. Although this study did not address the efficacy of single compared with combination antibiotic therapy it has been cited in influential articles that advocate combination therapy.⁹,¹¹

Antibiotic resistance

Randomised controlled trials of antibiotics often do not consider resistance as an outcome because it is difficult to measure, is beyond the time frame of most clinical trials, and usually affects patients in the community who are outside the trial.¹² The first two of these difficulties apply to cystic fibrosis, although the third is less relevant as emergence of a resistance within an individual patient has important long term consequences for that individual and his or her future treatment. We concluded in our review that the results concerning antibiotic resistance were preliminary and that further trials of single compared with combination antibiotic therapy were needed in cystic fibrosis. Given that our findings will add weight to the recommendations in support of combination therapy, it seems unlikely that such trials will now be possible.

In contrast with the stated advantages of combination therapy, usually with a β lactam and aminoglycoside, the theoretical arguments in favour of monotherapy with a β lactam include reduced costs, convenience, and a reduction in aminoglycoside related adverse effects. Respiratory exacerbations requiring intravenous antibiotic treatment may occur frequently from early childhood in patients with cystic fibrosis. Some clinics use regular courses of intravenous treatment every three months in patients with P aeruginosa.¹³ When considering these therapies in a chronic disease such as cystic fibrosis, consideration should be given not only to the short term outcomes, measurable at the end of treatment, but also to the long term outcomes, which may represent the cumulative effects of treatment over many years. The long term cumulative adverse effects of aminoglycosides are nephrotoxicity and ototoxicity. We searched Medline from 1966 to identify studies that investigated toxicity related to aminoglycosides in patients with cystic fibrosis.

Long term adverse effects

We found no longitudinal studies reporting on hearing or renal function in relation to cumulative doses of aminoglycoside. We did, however, find one cross sectional study on this subject.¹³,¹⁴ This study was from the first major cystic fibrosis centre to advocate intravenous β lactam and aminoglycoside every three months in patients with P aeruginosa.¹³ The study included 46 patients, with a mean of 20 cumulative courses of aminoglycoside. Nearly 40% of the patients had a creatinine clearance below normal, although there was no overt chronic nephrotoxicity. Two patients had a chronic hearing deficit that may have been drug related, and no chronic vestibulotoxicity was reported. A more recent study, which did not relate toxicity to cumulative dose of aminoglycoside, reported a series of 43 adults with cystic fibrosis in whom pure tone audiometry was performed. Bilateral sensorineural hearing loss for high frequency sounds consistent with aminoglycoside induced ototoxicity was detected in seven.¹⁵ We found two case reports of acute hearing loss in patients with cystic fibrosis related to treatment with aminoglycoside.¹¹,¹⁶

We also examined the outcomes used to measure nephrotoxicity and ototoxicity. Some studies investigated aminoglycoside toxicity by measuring urinary excretion of β N-acetylglucosaminase as a marker of renal tubular damage and distortion product otoacoustic emissions as a marker of cochlear damage.¹⁴,¹⁷,¹⁸ Traditional methods for assessing renal function and hearing have been criticised for being insensitive and detecting only irreversible end organ damage. It has been suggested that monitoring of patients using urinary excretion of β N-acetylglucosaminase and distortion product otoacoustic emissions may detect subclinical toxicity related to aminoglycosides.¹⁴,¹⁷ However neither of these measures has been properly evaluated as surrogate markers for the definitive end points.

Inadequacies of randomised controlled trials

Numerous examples exist of the inadequacies of randomised controlled trials in assessing long term adverse effects of treatments in other chronic diseases. The use of inhaled corticosteroids in asthma has been associated with considerable benefits, but in children there are concerns about the long term effects, especially on growth. A Cochrane review investigated the effects of linear growth in children with asthma treated with inhaled beclomethasone.¹⁹ The review methods included a comprehensive search for randomised controlled trials, but only three were suitable for inclusion in the review. The longest study duration was 54 weeks. Although the results of the review found a significant decrease in linear growth over this time, it was unclear whether this would be sustained in the longer term, and the conclusions were therefore tentative. As childhood growth patterns are not uniform, a full evaluation of the effect of inhaled steroids on growth may need to consider inclusion of longer term, non-randomised studies.

The need to consider non-randomised studies for the assessment of adverse effects has recently been addressed in some Cochrane reviews. A systematic review of low dose oral corticosteroids compared with placebo or non-steroidal anti-inflammatory drugs in rheumatoid arthritis concluded that this therapy is highly effective.²⁰ In the randomised controlled trials included in this review, long term adverse effects were not adequately addressed in either the treatment or the control groups. Because of this the reviewers searched for further studies to investigate medium and long term adverse effects of low dose steroids. In this part of the review they included eight additional trials and two matched cohort studies. In both of the cohort studies an increase in the incidence of adverse events was shown in the steroid treated group compared with the comparison group. Although this review recognised the need to examine studies other than randomised controlled trials to investigate long term adverse events, this was not included in the initial objectives of the review, and the methods did not include adverse events in the clinical outcomes.

Systematic reviews and lifelong diseases

We are faced with several dilemmas. In our systematic review of several small, short term clinical trials we found an increase in antibiotic resistance in patients treated with single compared with combination intravenous antibiotics. We concluded that these findings require further investigation by high quality clinical trials. In this example experimentation may be considered inappropriate (because the outcomes of interest are far in the future) or impossible (because of the reluctance of clinicians to participate).²¹ The reluctance of clinicians to participate may be successfully overcome, and the exponential increase in randomised controlled trials in cystic fibrosis over the past 30 years suggests that this is happening.²² However, because of prevailing medical opinion, reinforced by the tentative evidence from the systematic review, it seems unlikely that such clinical trials will be considered acceptable. Even if they were possible, to answer important questions about the safety and efficacy of treatment strategies clinical trials should compare repeated courses of treatment over several years. This leads to a tension between evaluating short term outcomes, which indicate a need to change practice, and continuing the trial for sufficient follow up time to evaluate long term outcomes. With a chronic disease that lasts for the lifetime of the patient it is hard to accept that all the important questions about treatment interventions will be answered by clinical trials. To address these questions, the role of other study designs such as population based cohort and case-control studies and analyses of epidemiological databases should be considered.

Reviewing data from observational studies

Disease databases such as cancer registries, which collect routine data, are invaluable for the documentation of incidence and prevalence of diseases and for use as a sampling frame for further studies. More specialised disease databases, which prospectively collect a range of clinical data, have also been established. A comprehensive database for patients with cystic fibrosis has been in existence for some years in the United States and has recently been established in the United Kingdom.²³ As well as providing important information about disease demography, complications, and epidemiology, such databases can be used to capture information prospectively about patient outcomes and relate these to treatment interventions and environmental and other independent variables. However, to achieve this it is important that they are focused on specific clinical questions at conception so that relevant data—for example, adverse effects of treatments or clinical outcomes—can be collected prospectively.

Clinicians and patients need to be able to make informed judgments that balance the benefits and harms of treatments. Judgments are more easily made about short term effects by using randomised controlled trials, but the assessment of long term adverse effects requires a different study design. In making recommendations for primary research, rather than constantly advocating further randomised controlled trials, systematic reviewers should consider whether useful information could be obtained from well conducted observational studies, which may require less financial, clinical, and patient resources. Although the limitations of non-randomised comparisons are well recognised,²⁰ we suggest that to adequately evaluate both benefits and risks of treatment, systematic reviews of chronic or lifelong diseases should explicitly review data from long term observational studies. Future methodological research should consider robust ways of integrating these data into the, now traditional, systematic review of randomised controlled trials.

Figure.

Studies comparing β lactam antibiotic with β lactam and aminoglycoside combination. Outcome is number of patients with resistant strains of Pseudomonas aeruginosa