Comprehensive evaluations of the adverse effects of drugs: importance of appropriate study selection and data sources

Abstract

While systematic reviews and meta-analyses are at the top of the evidence hierarchy, most of the methodology has focused on assessing treatment benefit. Hence, we propose a structured framework for the initial steps of searching and identifying relevant data sources so that adverse effects can be evaluated in a comprehensive, unbiased manner. The unique methodological challenges stem from the difficulties of addressing diverse outcomes encompassing common, mild symptoms to rare, fatal events. Retrieval of the most appropriate studies should be specifically tailored to fit the nature of the adverse effects, according to the primary objective and study question. In our framework, the structure of the review takes different forms depending on whether the main aim is on scoping/hypothesis generation, or evaluating statistically the magnitude of risk (hypothesis testing), or clarifying characteristics and risk factors of the adverse effect. The wide range of data sources covering adverse effects all have distinct strengths and limitations, and selection of appropriate sources depends on characteristics of the adverse effect (e.g. background incidence and effect size of the drug, clinical presentation, time of onset after drug exposure). Reviewers need to retrieve particular study designs that are most likely to yield robust data on the adverse effects of interest, rather than rely on studies that cannot reliably detect adverse effects, and may yield ‘false negatives’. Type II errors (a particular problem when evaluating rare adverse effects) can lull us into a false sense of security (e.g. wrongly concluding that there was no significant difference in harm between drug and control, with the drug erroneously judged as safe). Given the rapid rate at which methodological improvements occur, this proposed framework is by no means definitive, but aims to stimulate further debate and discussion amongst the pharmacoepidemiological and systematic review communities to reach a common consensus on the best methods.

Background

Systematic reviews and meta-analyses are at the top of the hierarchy of evidence as they should be based on rigorous and reproducible methods for synthesizing a comprehensive dataset, and there exists some previous advice regarding methods for the systematic assessment of adverse effects [Loke et al. 2007]. In this article, we provide more detailed guidance on the key initial steps for researchers who may not yet be familiar with tackling the diverse and complex range of drug safety studies, but are nevertheless interested in conducting unbiased evaluations of adverse effects. Our primary focus is on the challenging task of obtaining a comprehensive dataset, with particular regards to selecting studies that are likely to yield valid results, as well as relevant data sources. Rather than considering single or selected studies in isolation, the systematic review needs to thoroughly appraise the available evidence, and depending on the nature of the data, can deploy statistical techniques to summarize overall estimates in a meta-analysis. However, most of the methodology for systematic reviews has been developed around the assessment of treatment benefit using randomized controlled trial (RCT) data, and indeed, systematic reviews of effectiveness far outnumber the reviews of harm [Papanikolaou and Ioannidis, 2004; Aronson et al. 2002; Ernst and Pittler, 2001a, 2001b]. This could lead to a potentially biased situation where treatment benefits are emphasized, but the adverse effects receive little mention [Loke et al. 2007].

To start with, we must emphasize the specific challenges in systematically assessing adverse effects, and the need to be flexible in tackling these hurdles. Unlike beneficial effects which often have a well-defined or agreed primary endpoint, the scope of adverse effects is vast, ranging from common, mild symptoms to very rare fatal events. Some adverse effects are predictable from the pharmacological properties and are easily monitored, while others are idiosyncratic and come ‘out of the blue’, thus making it almost impossible to systematically capture. No single review method can succeed in identifying and synthesizing evidence on such a diverse range of adverse outcomes and there is a need to tailor the methods to fit the nature of the adverse effects. Here, we outline a framework for selecting and retrieving data for inclusion in systematic reviews to help us obtain unbiased estimates on the risk of harm.

Framework for data synthesis according to objective of the review

In our framework, the starting point of the evaluation and subsequent synthesis of the adverse effects data are guided entirely by the study question. Is the main aim in obtaining a broad overview of the adverse effects in general of a particular drug? Alternatively, are there specific adverse effects of interest, and is the main aim in clarifying the numerical risk and/or the clinical characteristics of the adverse effects? These alternative approaches, and accompanying methods are outlined in Box 1.

Box 1.

Framework for a structured approach. What aspect of adverse effects are you interested in?

Approach A – scoping or hypothesis generating

Starting point: ‘I don’t really have specific adverse effects in mind yet, but would like to have a wide overview of a range of safety issues, and to keep an eye on any newly emerging problems.’

Aim: broad overview of safety problems associated with a particular intervention.

Data sources and searches: The first step would be to check product information sheets, regulatory authority bulletins and published case reports. Randomized controlled trials (RCTs) and observational studies can then be used to help identify any key adverse outcomes in published studies. In cases where the drug name does not have any corresponding indexing term, the subheading ae (adverse effects) may be ‘floated’ (searched attached to any indexing term, for example, ae.fs) combined with text words for the drug name.

These searches are broad rather than limited, as they are a scoping method to identify areas of concern.

Data interpretation: it should be possible to identify specific areas of interest or concern that could be addressed through Approach B and C. This may involve consultation with topic experts and patient groups to prioritize specific adverse effects for detailed evaluation.

Approach B – hypothesis testing

Starting point: ‘I have particular concerns about certain adverse events and want to assess whether these events are caused by the intervention, and estimate the magnitude of risk.’

Aim: To determine (usually through tests of statistical significance) whether a specific adverse effect is likely to be caused by the drug, or whether bias or confounding or mere chance is at play. This typically involves calculation of the magnitude of risk (relative risk, odds ratio, number needed to treat for harm) and degree of uncertainty (95% confidence intervals).

Data sources and search: Selection of the most appropriate study designs (RCT or observational study) may vary depending on characteristics of the specific adverse effect. If time and resources are limited, the simplest approach is to check all the relevant RCTs first and if no reliable estimates are available, then it would be sensible to proceed to observational studies. A more comprehensive and resource-intensive approach would aim to consider and compare findings from both study designs together [Singh et al. 2007b].

Data interpretation: Acceptance of a causal relationship is based on rejection of the hypothesis that the ‘observed difference in frequency of adverse events between the drug and a comparator is due to bias or confounding, or from chance alone’. This requires judgment about potential of bias and statistical precision (confidence interval) of the comparison i.e. is ‘like being compared with like’?

Approach C – clinical application, susceptibility factors, preventability

Starting point: ‘I am interested in specific adverse effects, and in addition to the statistical estimates of risk (as in Approach B above), I want to explore the clinical characteristics and risk factors for the adverse effect in greater detail.’

Aim: To determine whether occurrence of the adverse effect has any relationship with dose, duration of treatment, and characteristics of the patients (examples include age, gender and genetic predisposition). This can involve establishing a dose–time–susceptibility relationship, [Aronson and Ferner, 2003], or determining preventability, or markers for patients at high risk for the adverse effect [Aronson and Ferner, 2010].

Data sources: At its simplest level, this would involve assembling a cohort of published cases and/or spontaneous reports so that individual patient factors could then be analysed [Singh and Loke, 2006; Singh et al. 2007b].

A more comprehensive approach would be to search and retrieve observational studies that have formally estimated relative risk or odds ratios for the association of a particular risk factor with the adverse effect (e.g. toxicity of methotrexate with certain genotypes) [Lee and Song, 2010].

Data interpretation: specific risk factors for the adverse effect may become apparent, and this may guide clinical recommendations for avoiding or reducing harm, as well as providing a basis for developing and testing strategies (e.g. pharmacogenetic testing and other clinical monitoring methods) for minimizing risk.

Determining the selection criteria for relevant studies

Before proceeding, we need to outline an important threat to validity when selecting studies for inclusion in a review of adverse effects. Review methods for studies of effectiveness are focused on minimizing bias from the risk of type I errors so that we do not give useless remedies to patients from wrongly finding a significant benefit where none actually exists. In contrast, the opposite view is important for adverse effects, where we are most concerned about type II errors which can lull us into a false sense of security (e.g. wrongly concluding that there was no significant difference in harm between drug and placebo, and the drug erroneously judged as safe).

The sources of type II error stem from the difficulties in rigorously defining unexpected outcomes, inadequate monitoring and under-reporting, insufficient sample size or follow-up duration to measure rare events, exclusion of patients with risk factors for adverse events, and slicing up of adverse effects data into many subcategories (with few events in each subcategory) [Breau et al. 2010; De Vries and Van Roon, 2010; Pitrou et al. 2009; Loke et al. 2007; Loke and Derry, 2001]. Reviewers need to identify and retrieve the specific study designs that are most likely to yield robust data on the adverse effects of interest, rather than rely on studies that cannot reliably detect adverse effects, and may yield ‘false negatives’.

Selection of studies that are likely to be informative according to the nature and characteristics of adverse effects

Reliable detection and reporting varies with predictability of the adverse effect, for instance, based on past knowledge of recognized findings with similar drugs in the class, or a strong link with the mechanism of action of the drug. For example, bradycardia with beta-blockers, and cardiac arrhythmias with amiodarone are very well monitored in cardiovascular trials of these agents, because these adverse effects are expected, and RCTs or cohort studies would prove informative [Loke et al. 2004].

Unusual events, with striking or distinct clinical features are likely to be captured through spontaneous reporting, case reports or case series, be it within clinical trials or through regulatory authority pharmacovigilance systems. For instance, progressive multifocal leukoencephalopathy is a rare, severe viral infection associated with certain biologic agents [Boren et al. 2008]. Even nonsevere, reversible events may be captured through spontaneous reporting because of the distinctive features, such as corneal deposits with amiodarone [Loke et al. 2004]. Although spontaneous reports may provide a signal, more detailed information on the magnitude of associated risks of rare events is probably best sourced from case–control designs.

For quantitative analysis of relative risk, we need to consider three key aspects: typical background incidence of the adverse outcome in unexposed patients, onset (timing) of event relative to drug exposure, and anticipated magnitude of increase in risk with the drug. Table 1 provides guidance of which studies are most likely to provide reliable estimates of risk depending on the nature of the adverse effect.

Table 1.

Guide to choice of relevant study designs for systematic reviews of harm, depending on specific characteristics of the adverse event of interest.

Time course of AE	Probably occurs with short exposure or soon after exposure				Follows long-term exposure or long period after exposure
Background incidence of AE in unexposed patients with disease	Common		Rare		Common		Rare
Anticipated relative risk increase with intervention	↑↑	↑	↑↑	↑	↑↑	↑	↑↑	↑
Study designs most likely to yield useful data on frequency of the specific adverse effects	RCTs (both large or small)	RCT (large) or meta-analysis of smaller RCTs)	RCT (large) or meta-analysis of RCTs (even if small; then many needed). Additional info from nonrandomized studies may be necessary	Meta-analysis of all available RCTs. Additional info from nonrandomized studies may be necessary	Large long-term RCTs. Meta-analysis of all long-term RCTs, even if small	Meta-analysis of nonrandomized studies. Meta-analysis of all long-term RCTs, small or large	Nonrandomized studies. Meta-analysis of large long-term RCTs	Meta-analysis of nonrandomized studies

AE, adverse event; RCT, randomized controlled trial.

For instance, if the adverse outcome is a common complication of the disease that occurs even in unexposed patients, new adverse events are unlikely to be detected through spontaneous reporting systems as physicians and patients would not have suspected and reported the culprit drug, e.g. myocardial infarction with rosiglitazone in type II diabetes [Singh et al. 2007a]. Given that myocardial infarction is common and expected in patients with diabetes, controlled clinical data from RCTs and large pharmacoepidemiological studies would be the most useful sources.

However, the timing of the studies are also crucial as the detection and reporting of a particular adverse event may increase exponentially as the suspicion with a new signal grows with time. In this instance (e.g. the proliferation of published articles on cardiovascular adverse effects of thiazolidinediones) [Wang et al. 2010], earlier trials may not have provided substantial adverse effects data, but after a signal has been spotted, subsequent trials (with prespecified ascertainment of cardiac outcomes in their protocols) have yielded much more data [Komajda et al. 2010; Home et al. 2009]. Choice and availability of data source may therefore vary depending on how ‘hot’, important or controversial a particular topic may be at that time. Conversely, spontaneous reporting systems may be less effective in situations where the healthcare professionals and patients feel that the adverse effect is already well recognized, and do not take time to file the report.

The predictability and clinical presentation of the adverse effect is another major influence on the likelihood of the event being detected and reported. For instance, mild events which are typically seen in the general population (e.g. muscle ache with statins, cough with angiotensin-converting enzyme [ACE] inhibitors) are unlikely to be detected in nonrandomized studies that rely on pharmacoepidemiological databases of hospital admissions. The relative risk of these events is potentially measurable in RCTs or controlled prospective cohort studies, if the protocol actively seeks to ask specific questions regarding these symptoms.

The often unpredictable and diverse nature of adverse effects discussed above substantially increases the complexity surrounding three key areas of review methodology: study designs, data sources and search strategy. Here we summarize the types of data available and their individual limitations.

Data sources

Sources outside of the traditional scientific peer-reviewed literature include manufacturer-produced product information leaflets, as well as the websites and databases of regulatory authorities and pharmaceutical companies. See Table 2 for specific examples (online Supplementary Table 3 provides additional examples of other resources of adverse effects data).

Table 2.

Major sources of information on adverse effects.

Sources	Examples, and where to find them	Type of information
Product Information sheets	UK ABPI electronic Medicines Compendium (eMC), http://emc.medicines.org.uk	In the past, this typically had simple listings of a range of suspected adverse effects. More recent versions incorporate estimates of frequency and some have unpublished trial data.
Regulatory authorities bulletins and drug reviews	Regulatory Authorities:	Provides safety updates on new signals or topical issues in safety monitoring.
	• Canadian Adverse Reaction Newsletter (CARN), http://www.hc-sc.gc.ca/dhp-mps/medeff/bulletin/index-eng.php
	• Drug Safety Update, http://www.mhra.gov.uk/Publications/Safetyguidance/DrugSafetyUpdate/index.htm
	• FDA Medwatch, http://www.fda.gov/Safety/MedWatch/default.htm
	• Australian Medicines Safety Update, http://www.tga.gov.au/adr/aadrb.htm
	• Netherlands LAREB updates, http://www.lareb.nl/kennis/kwartaalberichten.asp
	FDA Drug Approval, http://www.accessdata.fda.gov/scripts/cder/drugsatfda/	Statistical and clinical pharmacology review of New Drug Applications by the FDA that may contain important unpublished data and analyses.
Pharmaceutical company study registers	Main portals for pharmaceutical companies:	Include data from unpublished studies, trials and phase IV postmarketing surveillance data.
	• Clinical Study Results Database, (http://www.clinicalstudyresults.org/)
	• International Federation of Pharmaceutical Manufacturers and Associations (IFPMA) clinical trials portal, (clinicaltrials.ifpma.org/)
	• Lead Discovery, (http://www.leaddiscovery.co.uk)
	Example of specific pharmaceutical company register:
	• GlaxoSmithKline clinical study register, http://www.gsk-clinicalstudyregister.com
Spontaneous reporting	Internet:	Provide details of spontaneous case reports. Databases and specific request services require a fee. Publicly available information on the websites can be difficult to search systematically or interpret, and requires considerable expertise in relational databases.
	• Adverse Drug Reactions Database, http://www.adverse-drug-reaction.net
	• Canada’s Adverse Drug Reaction Database, http://www.hc-sc.gc.ca/dhp-mps/medeff/databasdon/index-eng.php
	• UK Yellow Card scheme: Drug Analysis Prints (DAPs), http://www.mhra.gov.uk
	• Netherlands LAREB, http://www.lareb.nl/kennis/zoeksignalen.asp
	Databases:
	• DIOGENES: Adverse Drug Events Database for data from the Food and Drug Administration (FDA) US MedWatch service, (http://www.foiservices.com/brochure/diogenes.cfm).
	• PharmaPendium, http://www.info.pharmapendium.com/
	Request services:
	• FDA's Adverse Event Reporting System (AERS) and Spontaneous Reporting System (SRS), http://www.foiservices.com/brochure/ADR_search.cfm or http://www.ntis.gov/products/adverse.aspx or http://www.fda.gov/cder/aers/extract.htm
	• Vigibase Services: Uppsala Monitoring Centre (WHO) collects individual reports from 77 countries, http://www.umc-products.com/DynPage.aspx?id=10671
Bibliographic databases	• MEDLINE, http://www.nlm.nih.gov/
	• EMBASE, http://www.embase.com/
	• BIOSIS, www.biosis.org/
	• Derwent Drug File, http://scientific.thomsonreuters.com/products/drugfile/
	• International Pharmaceutical Abstracts (IPA), http://scientific.thomsonreuters.com/products/ipa/
	• Iowa Drug Information Service (IDIS), http://www.uiowa.edu/∼idis/idistday.htm
	• PASCAL, http://international.inist.fr/article21.html
	• Science Citation Index (SCI), http://scientific.thomsonreuters.com/products/sci/	Contain bibliographic details of published case reports, case series, epidemiological nonrandomised studies and randomized studies. Can be difficult to search effectively if there is a lack of adverse effects terms in the title, abstract and indexing.
	• TOXLINE (Toxicology Literature Online), http://toxnet.nlm.nih.gov/cgi-bin/sis/htmlgen?TOXLINE	Some specialist databases require a subscription fee.

Product information sheets

These give information on the pharmacokinetics and biological mechanism of the drugs, and thus allow certain adverse effects to be predicted or anticipated. For instance, beta-receptors regulate the heart rate and, therefore, a review of harm with beta-blocker drugs might look for evidence of slow heart rates.

Although product information sheets have historically provided a simple listing of adverse effects, more recent versions now incorporate estimates of frequency (e.g. rare, uncommon, or common), and some even have unpublished trial data from the manufacturer.

Limitations are that the content of product information sheets vary from country to country. We are not certain whether selection of the data and analysis is systematic (e.g. the relative risk is not always described nor are the comparators), and source data may not be available for replication by external independent analysts.

Regulatory authority and pharmaceutical company websites

There are regular updates published on regulatory authority websites (see Table 2 for examples), some of which include data from unpublished studies that the regulator is able to access. This is also true for pharmaceutical company study registers which can include trials, as well as phase IV postmarketing surveillance data (Table 2).

Limitations are that there is no simple or systematic method of searching these websites, and the source data may not always be referenced or made available for public scrutiny. The dataset may not have been independently peer-reviewed outside of the manufacturer or regulator.

Databases of spontaneous reporting collected by regulatory authorities

Detailed raw data on individual spontaneous reports is available for download in database formats without charge from the US Food and Drug Administration (FDA) and Canadian regulators, but considerable expertise in relational databases is required to properly search and interpret the reports. The Dutch and UK authority provides less-detailed formats that are searchable on drug name.

The nature of the adverse effects data varies amongst providers. Multiple database files containing patient characteristics, time course, causality assessment and outcome of reactions in individual patients are available from the FDA or the Canadian authority websites. In contrast, the UK Medicines Healthcare Regulatory Agency uses the Yellow Card system to provide a table of simple numerical estimates on the types and numbers of suspected adverse reactions linked to a particular drug, without giving any clinical information on individual case reports.

Analysis of spontaneous reports is typically focused on new safety concerns or signals and can sometimes provide a rough idea of possible frequency (if extremely high or low, or rare or a totally new pathology). However, there is no denominator or control group to allow proper quantitative assessment of risk. A number of statistical techniques (such as disproportionality analysis) have been used to overcome this issue for signal generation in the WHO Vigibase dataset, as well as the UK Yellow Card system [Giezen et al. 2010; Pariente et al. 2009].

Limitations are that these databases rely on spontaneous reports to the regulators (from the patients, healthcare professionals and industry sources), and under or selective reporting is a problem. Often the data is at an early stage, and clinical details may be incomplete. The format of the information varies considerably amongst the regulatory authorities, and considerable expertise is required in retrieving and interpreting the data. Assessment of causality in the individual patient can prove difficult.

There are also several types of adverse effects data arising from scientific peer-reviewed literature identified through electronic databases. These include a diverse range of study designs covering case reports and case series, nonrandomized studies and randomized trials.

Published case reports or case series

These can be used to help identify important or new safety signals that may merit further detailed review.

A simple way of retrieving case reports is through the use of databases such as PUBMED, EMBASE, etc. with the search terms ‘drug/ae’ (name of drug, generic and/or trade name followed by/ae) and publication type ‘case report’ (plus synonyms).

The nature of these reports usually focus on new or interesting cases, with some clinical or educational message, and have the advantage of making available clinical information on patient characteristics, feature of the interventions (dose, duration, etc.) and outcomes.

However, the main limitations include susceptibility to publication bias as the report has to pass through author, peer reviewer and journal editor to be prepared for publication [Loke et al. 2004]. Thus, it may capture rare, or severe, or more bizarre manifestations, but miss out on common or mild adverse events. There is considerable subjectivity in making the judgment of causality, and it only ‘proves’ the case for a particular individual, i.e. the intervention may have definitely caused the adverse event in that patient, but we cannot say the same for the next patient, or for the population at large.

Epidemiological nonrandomized studies to assess associated risk

A simple way of searching would be to use databases such as PUBMED, EMBASE, etc. based on terms such as ‘intervention/ae’ (name of drug, generic and/or trade name followed by/ae) AND terms for study design: cohort or case–control or observational or longitudinal or follow up or retrospective, etc.) [BMJ Clinical Evidence, 2011; SIGN, 2011; Fraser et al. 2006]. If there is a specific adverse effect of interest, it would be possible to search based on the adverse effect term (and its synonyms) together with intervention term.

These studies represent more formal evaluation (relative to case reports) of selected adverse effects, and may often be able to give a better quantitative estimate of risk. Nonrandomized studies are considered useful as they can capture rare events (with the case–control design), and can be more generalizable due to broader population coverage, with potential for longer follow up. The main advantage is that the adverse effect is often the primary outcome for the observational study, and the researchers attempt to define and capture events in a more rigorous manner than trials where adverse events are tertiary outcomes.

Limitations stem from the lack of randomization, which may lead to relatively higher threat of bias, particularly with confounding. The pivotal question is whether risk factors for adverse effects could have influenced prescribing, and requires judgment on the following:

• whether or not the adverse effect was known at time of prescribing: if not known, it cannot have influenced prescribing;

• whether risk factors for the adverse effect were known at time of prescribing (if not known in general, or not routinely assessed, it cannot have influenced prescribing);

• whether restriction of analysis to specific populations (e.g. persons without risk factors) and/or use of comparator drug with same indication could have reduced confounding [Schneeweiss et al. 2007]; the use of restriction is one that the data analyst can influence most, but can result in loss of generalizability;

• adequacy of adjustment (e.g. confounder score, propensity score, instrumental variable analysis) for confounding variables.

Randomized studies to assess causal relationship between drug and adverse event

A simple way of searching would be to use databases such as PUBMED and EMBASE with existing clinical trial search filters [Lefebvre et al. 2008; Haynes et al. 2005]. However, electronic database searches relying on screening titles and abstracts are potentially unreliable in such situations because adverse effects may not be mentioned in the title, abstract or keywords/indexing terms [Derry et al. 2001]. It is likely that full-text evaluations of all potentially eligible trials will be needed.

There are important issues in considering the type of information from RCTs. Randomization is the guarantee against average (or asymptotic) unbiasedness. This potentially provides adverse effects data with high level of internal validity and lack of confounding, but only if adequate monitoring of adverse effects is also in place. RCTs are particularly suited for adverse effects that are predictable, well recognized or anticipated based on the pharmacological mechanism, where it is likely that investigators would have detected and reported such events.

Limitations relate to the comprehensiveness of outcome reporting, adequate sample size and duration of trial follow up. Reports of adverse effects are often relegated to small sections of published manuscripts [Loke and Derry, 2001], and the search may have to be supplemented by evaluation of the pharmaceutical company trial register and regulatory authority assessments [Loke et al. 2009]. If an adverse event is not mentioned in a trial report, it cannot be known whether the investigators did not look for it, not record it, or did not present it, either because there was none or no difference, or because the difference in the trial was deemed ‘not significant’. The available trials may not yield reliable estimates of certain rare or unpredictable events (see Table 1).

Given the problems in selecting relevant studies for particular adverse effects, a wide variety of study designs (rather than just RCTs) may need to be reviewed in order to address specific questions on unexpected rare adverse events. The diversity of studies creates major challenges for the process of data interpretation and synthesis, as well as the assessment for risk of bias. While specific advice on these other steps are beyond the scope of this article, we nevertheless recommend that authors embarking on a systematic review should carefully plan how they tackle a varied range of studies, perhaps by looking at examples of published reviews. Moreover, a recent methodological review has also found that optimal search strategies are not yet available for identifying studies of adverse effects, and further empirical work is needed on the methods proposed here [Golder and Loke, 2010].

Future directions

This proposed framework is a work in progress, and does not aim to be definitive. We are certain that methods of systematically assessing adverse effects will continue to rapidly progress, and we encourage debate and discussion amongst the pharmacoepidemiological and systematic review communities. We hope that this article will start a much needed debate towards reaching a common consensus on the best methods of obtaining a comprehensive unbiased evaluation of adverse effects.

Conclusions

In this paper, we have proposed a structured framework for study selection when undertaking comprehensive evaluation of adverse effects, where the choice of strategy depends on the main objective and study question. We have discussed the strengths and limitations of the various data sources for adverse effects, and propose that choice of data source should be based on the nature of the adverse effect. If there is sparse data for quantitative analysis of adverse effects, we believe that the primary focus should be on rigorous assessment of the potential for type II errors and the level of uncertainty manifest within the 95% confidence intervals, rather than deriving potentially biased conclusions regarding ‘no significant harm’.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Conflict of interest statement

Some of the work described here was undertaken by Su Golder as part of an UK Medical Research Council Health Services Research fellowship. The views expressed in this presentation are those of the authors and not necessarily those of the MRC.