Evidence-based Assessment of Well-Established Interventions: The Parachute and the Epidural Hematoma

Abstract

Background:

The methods of evidence-based medicine are a relatively recent development in the understanding of clinical practice. They are criticized as not providing support for interventions long held to be highly effective based on experience that predated the availability of evidence-based analysis.

Objective:

To determine if the methods of evidence-based medicine can be successfully applied to interventions established before those methods were developed.

Methods:

Systematic review of English language literature on the natural history and treated prognosis of acute epidural hematoma and analysis of existing data on mortality associated with parachute use.

Data Sources included Medline, Old Medline, Science Citation Index, British and United States Parachute Associations, and Federal Aviation Administration and National Transportation Safety Board databases (both of the United States). Also included were national databases reporting mortality and total number of parachute uses.

Results:

The estimated mortality of falling from an airplane with an ineffective parachute is 74% (69–79). Mortality associated with effective parachute deployment is between 0.0011 and 0.0017%. For acute epidural hematoma, estimated mortality is 98.54% (95.1 – 99.9) without treatment and 12.9% (10.5 – 15.3) with treatment. The Number Needed to Treat to prevent one death for the parachute is estimated to be 1.35 (1.27–1.45) and for epidural hematoma 1.17 (1.13 – 1.22) (95% binomial confidence intervals in parentheses).

Conclusion:

The methods of evidence-based medicine are robust and can deal with interventions of great face validity and those considered well established before such methods were well developed. We propose initial criteria for evaluating the quality of evidence supporting long established interventions.

One of the vexing problems of the evidence-based approach to neurosurgical practice is the difficulty in dealing with well-established interventions that became a standard part of practice before the techniques of evidence-based medicine were well established. Resistance to the use of evidence-based medicine techniques, resource limitations and the complexities of technical development continue to create obstacles to early rigorous evaluation of new therapies. New interventions may be in widespread use before an organized effort at acquiring high quality evidence of effectiveness and safety is begun. Efforts to develop evidence-based guidelines for such interventions can be difficult, as is the well documented case for the treatment of epidural hematoma for which the guidelines group felt it necessary to abandon the accepted paradigm for labelling strength of recommendations.^{1, 2} There may be true uncertainty about the role of such an intervention. There may be resistance to the use of such an intervention from non-neurosurgical evidence-based practitioners. Finally, the fact that interventions exist that became well-established before the rules of evidence were well understood is often used by critics of evidence-based medicine as “evidence” that rigorous application of evidence-based techniques is not necessary and might even be detrimental if used to cast doubt upon standard practice.

This latter issue is exemplified in the 2002 article by Smith and Pell entitled “Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomized controlled trials”.³ In poking fun at those who demand randomized controlled trial evidence to support therapeutic intervention they use the parachute as an example of an imperfect intervention for which no randomized controlled trial evidence exists, sensibly enough suggest that no-one is likely to volunteer to participate in such a trial and claim that only two options exist to handle this situation: to “accept that, under exceptional circumstances, common sense might be applied when considering the potential risks and benefits of interventions” or to “continue our quest for the holy grail of exclusively evidence based interventions and preclude parachute use outside the context of a properly conducted trial”. We suggest that there is a third option: to develop rigorous standards that allow historical observational evidence to be accepted within the framework of evidence-based medicine. Such methods would answer the criticism that strict adherence to evidence-based principles discards much of the progress made with observational science and process improvement techniques over the past centuries while identifying those “well-established” interventions for which real uncertainty still exists and which merit further rigorous investigation.

The parachute example is useful in developing such standards because it represents a well-known situation in which the acceptance of the intervention has great face validity. Any technique designed to validate a “well-established” therapeutic neurosurgical intervention should provide support if the intervention has effectiveness comparable to that of the parachute. The surgical treatment of acute epidural hematoma is frequently given as an example of a highly effective neurosurgical intervention for which no randomized controlled trial evidence exists and for which no neurosurgeon would propose such a trial.

Therefore, we undertook to investigate the strength of the evidence supporting the perceived effectiveness of the parachute and compare it to the evidence supporting the effectiveness of surgical treatment of acute epidural hematoma.

Methods

The natural history of jumping from airplanes

An electronic search of Medline (1966 – 2008), Old Medline (1948 – 1965) and the Science Citation Index (1975 – 2008) was completed for “parachuting”, “skydiving” or “parachuting fatalities”. Also, the joint Federal Aviation Administration (FAA) and National Transportation Safety Board (NTSB) database of parachuting events in which a significant injury or death occurred was reviewed.⁴ The NTSB requires that all civilian incidents involving death or serious injury be reported. Any report where a parachutist had a free fall which resulted in impact with the ground was included. Reports of both intentional and unintentional descents without effective parachutes were included. Any parachute failure by either the equipment or the parachutist was included as long as the event resulted in a free fall to the ground. Additional information was obtained from the survey of Hart, et. al. identified in the literature search.⁵

The effectiveness of parachutes

We reviewed existing literature on the fatality rate of parachute jumps from two national organizations, the United States Parachuting Association and the British Parachute Association. They maintain large databases of recreational parachute jumps.^{6, 7} Absolute risk reduction, relative risk reduction and number needed to treat and their binomial confidence intervals were calculated for estimates. ⁸

The natural history and treated prognosis of acute epidural hematoma

An electronic search of Medline and Old Medline was conducted using “epidural hematoma” and “extradural hematoma.” The references of these articles were then investigated along with classic neurosurgical texts to complete the review of the literature. Articles had to describe sample population, overall mortality, and have at least 20 patients analyzed. Articles limited by design to only one possible outcome (all survived or all died) were excluded. Walters’ criteria for levels of evidence for studies of prognosis were used to evaluate each prognostic study.⁹ Each article was assessed for the following criteria and given an appropriate level of evidence:

• –Patients are assessed at a uniform time in their disease
• –Patients are observed prospectively for a designated period
• –Patient outcomes are measured definitively and reliably
• –Patients are part of a continuous or defined cohort of at least 25 patients
• –Patients are examined for extraneous prognostic variables

Walters proposes that articles meeting all five criteria are designated as Level 1 evidence, four out of five criteria result in a Level 2 classification, and all others are Level 3.⁹

We elected to truncate data collection after examining the literature through 2008. We found no data that could be added to the “no treatment” group after 1988. The trend for surgical treatment was well established with data collected through 1999 and the next 9 years did not add any useful data. Given that the surgical treatment of epidural hematoma in the neurologically deteriorating patient is a well-established principle, and this analysis is aimed at assessing the ability of evidence-based medicine tools to assess well established interventions and therefore primarily dependent on historical data, we determined that additional data would not add materially to the conclusions.

Because not all of the studies had both a treatment and a control arm, traditional meta-analysis approaches were not possible. Instead, study mortality was modeled using a generalized linear mixed model, with logit link and binomial family distribution, with fixed effects for treatment, mean-centered year and their interaction, and a random effect of study to account for heterogeneity among studies. When treatment group was the only fixed effect in the model, then the study random effect variance was 0.8512 on the logit scale (95% confidence interval: 0.5122, 1.6868), which was significantly greater than zero, indicating heterogeneity among studies. When study year was added to the model, and its coefficient was allowed to differ by treatment, then the random effect variance decreased by nearly two-thirds, to 0.2643 (95% CI: 0.1460, 0.6176), but the remaining variance was still significantly greater than zero, indicating heterogeneity among studies in addition to the time trends. The analysis was carried out using the GLIMMIX procedure in SAS 9.2.

The comparative effect of treatment was estimated for 1988, the latest year for which both treated and untreated studies were available, since the effect of treatment varied significantly over time. Bootstrap re-sampling with replacement from the original dataset was used to obtain means and 95% confidence intervals: the original dataset was re-sampled with replacement 1,000 times, and the GLMM modeling was repeated on each re-sampled dataset. Means and 95% confidence intervals were obtained from the 1,000 sets of results for the 1988 model-predicted mortality in the treatment group and in the no-treatment group, as well as for the absolute risk reduction (ARR), the relative risk (RR), the relative risk reduction (RRR) and the number needed to treat (NNT).

The study results were also summarized using forest plots. Since the effect of treatment varied significantly over time, the summary mortality for the forest plots was calculated for the last year in which relevant studies were done: 1988 for untreated studies and 1999 for treated studies. The forest plots were made using the forestplot routine from the rmeta package in R 3.0.1.

Results

The natural history of jumping from airplanes

We could not identify data regarding the mortality of falling from an airplane without any form of parachute. Therefore, we rely on the data regarding the likelihood of surviving a fall from an airplane with a parachute (with the implication that death implies parachute failure). The rate thus derived may significantly underestimate the actual mortality of unprotected free-fall. Hart and Griffith in 2002 and 2006 used data from the United States Parachute Association which maintains a database of parachuting fatalities recorded on a voluntary basis. They noted 241 fatalities from 1993–1999 and 125 from 2000–2004.⁵ Their classification indicated that 82 percent of parachuting fatalities were caused by human error and not related to weather conditions or equipment error. These data are not identical to the data gathered from the FAA database because of the difference in techniques in data collection (voluntary reporting and survey, respectively). They do not provide a denominator for calculating a mortality rate. However, using the FAA/NTSB database we can provide enough evidence to show a mortality rate associated with falling from an airplane with an ineffective parachute. Of the 340 events from 1978 to 2007 that were included from the FAA database, there were 252 deaths with 88 injuries of varying severity giving an overall mortality of 74% (95% binomial confidence interval 69% – 79%).

The effectiveness of parachutes

Hart and Griffith quote a finding of 34 fatalities in 2 million recreational parachute jumps in 2002 for an estimated rate of 1.7 per 100,000 jumps or 0.0017 percent.⁵ Seventeen years (1990 – 2006) of data from the British Parachute Association documented a rate of 1.1 fatalities per 100,000 jumps or 0.0011 percent. ⁶

The natural history of acute epidural hematoma

Because the current treatment of epidural hematoma was developed before the era of CT and MR scanning, for the purposes of this analysis we have excluded clinically undetectable epidural hematomas from consideration. This excludes the unoperated patients of Cook¹⁰ and Servadei¹¹ who were selected for non-operative treatment because of CT scan findings suggesting small hematomas (undetectable before CT became available) and lack of neurologic deterioration. Deteriorating patients were operated upon in their studies. In all of the series reported before CT scanning was available, all known cases of epidural hematoma were included and in the two other post-CT series with unoperated patients^{12, 13} all unoperated patients were deemed by their surgeons to be moribund upon admission to the neurosurgical service. The data are shown in Table 1.

Table 1:

Reports of Natural History of Epidural Haematoma

Author	Year	Reference	Treatment	N	Died	l95ci	Mortality	u95ci	Level of evidence
Jacobson	1885	15	no	71	61	0.76	0.859	0.93	3
vonBergman	1927	16	no	99	83	0.75	0.838	0.90	3
Munro	1941	60	no	6	6	0.54	1.000	1.00	1
Lewin	1949	21	no	3	3	0.29	1.000	1.00	2
Rowbotham	1954	23	no	1	1	0.02	1.000	1.00	3
McKissock	1960	61	no	9	8	0.52	0.889	1.00	1
McLaurin	1964	25	no	2	2	0.16	1.000	1.00	2
Gallagher	1968	62	no	45	45	0.92	1.000	1.00	2
Heiskanen	1975	63	no	3	3	0.29	1.000	1.00	1
Phonprasert	1980	64	no	1	1	0.03	1.000	1.00	1
Yue	1983	65	no	4	4	0.40	1.000	1.00	2
Dan	1986	66	no	16	16	0.79	1.000	1.00	3
Simpson	1988	67	no	2	2	0.16	1.000	1.00	1

l95ci = lower 95 percent binomial confidence interval

u95ci = upper 95 percent binomial confidence interval

Jacobson in 1885 reported a large untreated series with 86% mortality.¹⁴ A large study by vonBergmann cited in Rose and Carless in 1927 also showed an untreated mortality rate of 86%.¹⁵ Recent smaller series of untreated cases located in our review have untreated mortality rates approaching 100%. There is a small but statistically significant increase in the mortality of untreated epidural hematoma over time (p=0.0433). The model-estimated mortality rate for untreated acute epidural hematoma in 1988 is 98.54 percent (95% confidence interval 95.1% – 99.9%). ^14–20 The data are plotted in Figure 1.

Figure 1.

Forest plot of Mortality from Studies of Untreated Epidural Hematoma. The mortality for each study is indicated by a square, with 95% exact binomial confidence intervals indicated by horizontal lines. The diamond indicates the model-predicted mortality and 95% confidence interval for no-treatment studies in 1988.

The effectiveness of surgical treatment of acute epidural hematoma

Published series of surgically treated epidural hematoma have shown a steady decline in mortality from 1938 to the present rate of approximately ten percent^{10–13, 16–48} (Table 2 and Figure 2). Assessed according to Walters’ criteria for quality of studies of prognosis, the epidural hematoma data was of good quality (Table 3). There is a statistically significant reduction in the mortality of treated epidural hematoma over time (interaction p=0.0005). The data and model-fitted trend lines are shown in Figure 3. The model-estimated mortality for treated epidural hematoma in 1988 is 12.9 percent (95% confidence interval 10.5% – 15.3%).

Table 2.

Reports of the Treated prognosis of Epidural Haematoma

Author	Year	Reference	Treatment	N	Died	l95ci	Mortality	u95ci	Level of Evidence
Pringle	1938	22	yes	33	22	0.48	0.667	0.82	3
Munro	1941	68	yes	38	20	0.36	0.526	0.69	1
Lewin	1949	21	yes	26	9	0.17	0.346	0.56	2
Bradley	1951	22	yes	27	15	0.35	0.556	0.75	3
Rowbotham	1954	23	yes	27	17	0.42	0.630	0.81	3
Hooper	1959	24	yes	83	19	0.14	0.229	0.33	1
McKissock	1960	69	yes	116	26	0.15	0.224	0.31	1
McLaurin	1964	25	yes	45	17	0.24	0.378	0.53	2
Gallagher	1968	70	yes	122	48	0.31	0.393	0.49	2
Jamieson	1968	71	yes	167	26	0.10	0.156	0.22	1
Weinman	1968	72	yes	155	31	0.14	0.200	0.27	1
Heiskanen	1975	73	yes	77	10	0.06	0.130	0.23	2
Kvarnes	1978	34	yes	132	30	0.16	0.227	0.31	1
Mendelow	1979	74	yes	83	14	0.10	0.169	0.27	2
Phonprasert	1980	75	yes	137	22	0.10	0.161	0.23	1
Yue	1983	76	yes	56	17	0.19	0.304	0.44	2
Bricolo	1984	77	yes	107	5	0.02	0.047	0.11	1
Reale	1984	78	yes	66	12	0.10	0.182	0.30	1
Seelig	1984	79	yes	51	21	0.28	0.412	0.56	1
Dan	1986	80	yes	110	16	0.09	0.145	0.23	3
Baykaner	1988	81	yes	95	9	0.04	0.095	0.17	1
Cook	1988	11	yes	91	9	0.05	0.099	0.18	2
Haselsberger	1988	43	yes	60	15	0.15	0.250	0.38	1
Rivas	1988	82	yes	161	19	0.07	0.118	0.18	1
Servadei	1988	83	yes	135	19	0.09	0.141	0.21	1
Simpson	1988	84	yes	107	17	0.10	0.159	0.24	1
Poon	1991	85	yes	104	9	0.04	0.087	0.16	1
Jones	1993	86	yes	366	61	0.13	0.167	0.21	2
Kuday	1994	50	yes	115	12	0.06	0.104	0.18	1
Jamjoom	1997	52	yes	120	15	0.07	0.125	0.21	2
Lee	1998	87	yes	200	13	0.04	0.065	0.11	2
Wester	1999	88	yes	83	1	0.00	0.012	0.07	2

Figure 2.

Forest plot of Mortality from Studies of Treated Epidural Hematoma. The mortality for each study is indicated by a square, with 95% exact binomial confidence intervals indicated by horizontal lines. The diamond indicates the model-predicted mortality and 95% confidence interval for treatment studies in 1999.

Table 3:

Quality of Epidural Haematoma Prognosis Studies

Levels of Evidence (Walters’ Criteria) 39 total articles

Level of Evidence	Not Treated		Treated
	N	%	N	%
One (all five criteria)	5	38	17	53
Two (4 of 5 criteria)	4	31	11	34
Three (3 or fewer criteria)	4	31	4	13

Some articles report both treated and untreated patients. Therefore the row and column totals exceed the number of articles in each category.

Figure 3.

Epidural Hematoma Mortality by Treatment. The triangles mark the no-treatment studies, the circles mark the treatment studies, the solid lines are the predictions of the generalized linear mixed model, and the dotted lines are the 95% confidence intervals for the model predictions.

Treatment Effect

The absolute and relative risk reductions and number-needed-to-treat for the parachute and surgical evacuation of acute epidural hematoma are shown in Table 4. They are remarkably similar (absolute risk reduction of 74% for the parachute and 85.6% for hematoma evacuation, NNT (95% confidence intervals) 1.35 (1.27 – 1.45) and 1.17 (1.13 – 1.22) respectively), and document very large treatment effects.

Table 4:

Treatment Effect for Parachutes and Epidural Haematoma Evacuation

	Parachute	Epidural Hematoma (for 1988)
Mortality without Intervention	74% (69 – 79)	98.5% (95.1 – 99.9)
Mortality with intervention	0% (.0011 – .0017)	12.9% (10.5 – 15.3)
ARR (absolute risk reduction)	74% (69 – 79)	85.6% (81.8 – 88.5)
RRR (relative risk reduction)	99.99%	86.9%
NNT (number needed to treat)	1.35 (1.27 – 1.45)	1.17 (1.13 – 1.22)

ARR = mortality without intervention – mortality with intervention

RRR = (mortality without intervention – mortality with intervention) / Mortality without intervention

NNT = 1 / ARR

Ranges are 95% confidence intervals except for Mortality with intervention for parachute which lists the two estimates available from large data databases

Discussion

Jumping out of an airplane without an effective parachute has a mortality rate similar to that of the five year mortality of lung and esophageal cancer or the acute mortality from epidural haematoma.^{49, 50} Such very high mortality rates create exceptional opportunities to demonstrate the effectiveness of therapeutic intervention because the outcome of interest is highly objective and readily verifiable.

We do not believe that it is necessary to rely solely on “common sense” to make the determination that the untreated prognosis of a condition is so uniformly bad and the intervention so uniformly good that the value of the intervention is obvious. Under these circumstances it is relatively easy to demonstrate a homogeneous objective outcome. What is lacking are agreed standards for the degree of homogeneity and the magnitude of treatment effect large enough to overcome any reasonably expected form of bias.

The question of whether randomized clinical trials (RCT) are necessary to provide sufficient evidence to incorporate an important new treatment into practice is not a new one. The ascendency of the RCT in the evaluation of new treatments that has characterized the last three decades has been based predominantly on this emphasis on study quality. While it is the quality rather than the quantity of evidence that drives the evidence hierarchy of evidence-based medicine ⁵¹, once quality has been assessed the size of treatment effect has been widely recognized as an important factor in assessing the likelihood that unidentified bias has affected the results in an important way.^52–54

Glasziou and colleagues reviewed the question of when RCTs might be unnecessary.⁵⁵ They proposed rules of thumb: “(a) that the conventionally calculated probability of the two groups of observations coming from the same population should be less than 0.01 and (b) that the estimate of the treatment effect (rate ratio) should be large. In our examples it was at least 20.” Because of the heterogeneity in reporting of the epidural hematoma treatment series, it is difficult to calculate a “rate ratio” for these data. Pereira, et. al., used the GRADE definition of a “very large treatment effect” as a relative risk ratio of >5. ^{52, 56} In the case of the parachute and the epidural hematoma, the relative risk ratios for death were 0.74/0.0015 = 493 (parachute) and 0.985/0.129 = 7.6 (epidural hematoma).

The problem of relying on large treatment effects to overcome bias sufficiently to establish efficacy continues to be a matter of debate. Cook et. al. provide an interesting example of a convincing scenario.⁵³ La Rochelle and Julien estimate the risk ratio in favor of handwashing described by Semmelweiss to be 5.36.⁵⁴ Pereira, et. al., performed an automated study of treatment effect size in the Cochrane Database of Systematic Reviews.⁵⁶ They concluded that most large treatment effects are found in small studies, that those treatment effect estimates tend to shrink as more studies are done, and that well validated large effects are uncommon and rarely pertain to fatal outcomes.

These investigations support the concept that, so long as the underlying observations are of reasonable quality, very large treatment effects may be sufficient to outweigh any practically conceivable bias and can be considered to provide sufficient evidence to support the inclusion of the tested intervention into clinical practice. Such determinations should be rare. While preliminary standards have been proposed, they are not yet widely accepted. We hope to add to the evidence base supporting the development of such standards.

We suggest that the parachute and the acute epidural hematoma are examples that have great face validity in beginning to establish these criteria. We propose the following criteria as a starting point, expecting further refinement as more examples are examined.

1. The outcome of interest should be serious and unambiguous (i.e. subject to very little intra- or inter-observer error). Death is the prototype outcome that would qualify. No subjective outcome can provide definitive evidence for this type of intervention assessment.

2. The (untreated) natural history of the condition in question should result in the outcome of interest in a very high (we suggest 70 percent) proportion of subjects. Multiple observational series should confirm the natural history.

3. The treated prognosis of the condition in question should document a reduction in the endpoint of interest to very low levels. Multiple observational series should confirm the treated prognosis and any shift over time in either natural history or untreated outcome or in definition of the disease should be minimal in comparison to the treatment effect. In the case of epidural hematoma, there is a progressive reduction in mortality of the treated patients. However, there is no indication that there is a decline in the mortality of untreated patients (excluding, as we have done, those with clinically undetected hematomas identified by CT or MRI scans). The binomial confidence intervals around the mortality rates do not overlap with those of the natural history estimates even at the time of the earliest reports of treated series of patients.

As a result, we have chosen to focus on the absolute risk reduction rather than the relative risk reduction as a measure of magnitude of treatment effect sufficient to overcome conceivable but unmeasured bias. The proposed degree of confidence in the decision to accept the conclusions of the accumulated evidence would be based on the overall reduction in objective outcome:

High Confidence: NNT < 1.43 (Absolute risk reduction >= 70%)

Moderate Confidence: NNT < 1.67 (Absolute risk reduction >= 60%)

Low Confidence: any other reduction in outcome

The confidence intervals around the NNT estimates should be sufficiently small that the categories do not overlap and do not overlap with a NNT of 2.0 (equivalent to a 50% ARR). We propose absolute risk reduction as the measure of treatment effect because relative measures (relative risk reduction, odds ratios, hazard ratios) can represent small absolute risk reductions as large treatment effects. Small absolute risk reductions may be subject to greater risks of measurement error and bias. While this may be overcome by very large sample sizes or extraordinary care in measurement, such precautions ordinarily require a prospectively planned trial which is not the target of this proposal.

It is useful to ask the question: “how do other neurosurgical procedures widely held to be highly effective fare when assessed by these criteria?” It turns out to be very difficult to answer this question because there are few procedures with such large treatment effects and many of them do not have the necessary natural history available to estimate absolute risk reduction.

We examined the extracranial-intracranial bypass procedure. It was widely practiced until the EC-IC bypass trial⁵⁷ failed to demonstrate useful therapeutic benefit. Unfortunately, the natural history of the conditions for which it was proposed was not well documented at the time.⁵⁸ The review of Samson and Boone⁵⁹ suggests that patients presenting with TIA or RIND treated with EC-IC bypass had TIAs abolished 85 – 95 percent of the time and 3–4 percent suffered a stroke during variable follow-up. They quote literature suggesting stroke rates of 15 to 40 percent in medically treated patients (but acknowledge that the comparison is not valid). Even if one accepted these figures, the absolute risk reduction from 40% to 3% (37%) would not meet our criteria for being sufficiently large to avoid further study. Despite the protestations of some advocates of EC-IC bypass, the procedure clearly required further study in a controlled paradigm.

A more recent possible example is the Wingspan endovascular stent. The single arm Wingspan study⁶⁰ suggested a stroke rate of 7.1% compared to similar historical controls with a rate of 11 to 14%. Again, the absolute risk reduction is so low that formal controlled studies were required (and, in fact, were terminated early because of unexpectedly high post-procedural stroke rates).⁶¹

There is widespread acceptance of the value of gross total resection in the treatment of glioblastoma. The estimates of effect on survival provided in a recent analysis of the SEER database⁶² are dependent on time at which survival is assessed and what treatment the control group received. Estimated from the Kaplan Meier plot reported in the Zinn study, the absolute risk reduction for gross total resection compared to no surgery is 32% at 10 months, 13% at 20 months, and 6% at 30 months. Corresponding ARR for gross total resection compared to subtotal resection are 6, 5, and 3 percent. None would meet our criteria.

Conclusion

The methods of evidence-based medicine address many questions about the practice of medicine. The principles of objective and reliable observation can be extended to provide rigorous criteria for evaluating the value of interventions established before the principles of evidence evaluation were well understood. The development and validation of such criteria can help to avoid inappropriate concern about the value of such interventions and unnecessary additional clinical research while also identifying interventions for which the evidence base requires more research. However, because such evaluations are frequently open to concerns about unmeasured bias, there must be reasonable assurance about the quality of the data and the observed treatment effect should be very large, so large that it would overcome any conceivable bias affecting the data. We propose that absolute risk reduction as a measure of treatment effect size is the most appropriate measure and that it should exceed 70% in order for a conclusion of high confidence to be reached. This is a very conservative criterion, but we believe that that is appropriate. Avoiding carefully planned scientific evaluation of new interventions should be a rare event. Additional examples of such interventions will need to be examined to more definitively establish these criteria.

Box 1.1: Critically appraising review articles.

• ✔Was the review question clearly defined in terms of population, interventions, comparators, outcomes and study designs (PICOS)?
• ✔Was the search strategy adequate and appropriate? Were there any restrictions on language, publication status or publication date?
• ✔Were preventative steps taken to minimize bias and errors in the study selection process?
• ✔Were appropriate criteria used to assess the quality of the primary studies, and were preventative steps taken to minimize bias and errors in the quality assessment process?
• ✔Were preventative steps taken to minimize bias and errors in the data extraction process?
• ✔Were adequate details presented for each of the primary studies?
• ✔Were appropriate methods used for data synthesis? Were differences between studies assessed? Were the studies pooled, and if so was it appropriate and meaningful to do so?
• ✔Do the authors’ conclusions accurately reflect the evidence that was reviewed?

Centre for Reviews and Dissemination, University of York, 2008