Key concepts for informed health choices. 3.1: evidence should be relevant

Make well-informed choices

What to do depends on judgements about a problem, the relevance of the available evidence, and the balance of expected benefits, harms and costs.

The first step in making decisions about treatments is being clear about what your problem or goal is and what the options are. Only then can you make judgements about the relevance of the available evidence from treatment comparisons, including the relevance of:

• the outcomes that were measured,

• the participants in the comparisons,

• the treatments that were compared and

• the circumstances in which the treatments were compared.

In this essay, we explain those considerations.

Once the trustworthiness of the available evidence has been assessed (as explained in essays 2.1–2.4) and its relevance (as explained in this essay), well-informed decisions about what to do depend on judgements about whether the expected advantages outweigh the expected disadvantages, including consideration of:

• the benefits, savings, harms and costs of acting or not,

• the baseline risk or severity of the symptoms,

• the importance of each advantage and disadvantage,

• the certainty of the evidence and

• the need for further fair comparisons.

We explain these considerations in the next essay (3.2).

Be clear about what the problem or goal is and what the options are

Good decisions depend on correctly identifying the problem and considering an appropriate set of options to address the problem. For personal health choices, this means starting with a correct diagnosis (or assessment of risk) and then identifying the treatments that are available. For public health and health system policy decisions, this means describing the problem correctly and identifying the policy options relevant for that problem. Changing how a problem is framed can lead to different options for addressing it.

The following passage from Archie Cochrane’s autobiography is a striking illustration of the importance of correctly identifying the problem and considering appropriate options. ¹ He recalls an event when he was a doctor in a German prisoner of war camp.

The Germans dumped a young Soviet prisoner in my ward late one night. The ward was full, so I put him in my room as he was moribund and screaming and I did not want to wake the ward. I examined him. He had obvious gross bilateral cavitation and a severe pleural rub. I thought the latter was the cause of the pain and the screaming. I had no morphia, just aspirin, which had no effect. I felt desperate. I knew very little Russian then and there was no one in the ward who did. I finally instinctively sat down on the bed and took him in my arms, and the screaming stopped almost at once. He died peacefully in my arms a few hours later. It was not the pleurisy that caused the screaming but loneliness. It was a wonderful education about the care of the dying. I was ashamed of my misdiagnosis and kept the story secret.

Consider the relevance of the outcomes measured in the research

A fair comparison may not include all outcomes – short- and long-term – that are important. Patients, professionals and researchers may have different views about which outcomes are important. For example, studies often measure outcomes, such as heart rhythm irregularities, as surrogates for important outcomes, like death after heart attack. The effects of treatments on surrogate outcomes often do not provide a reliable indication of the effects on outcomes that are important. Similarly, short-term effects may not reflect long-term effects.

Despite dozens of randomised trials since the introduction of the first oral agent for treating type 2 diabetes, it has remained uncertain if any of those medicines favourably affects outcomes that are important to people, including morbidity, mortality and quality of life. ² A key reason for this is that the trials have focused on glucose control measured with laboratory tests rather than on outcomes that are important to people with diabetes. Unfortunately, those laboratory tests (glycosylated hemoglobin, type A1C – glycosyl HbA1C) are not a reliable indicator of outcomes that are important to people with type 2 diabetes.

It is sometimes important to consider outcomes that are important to other people besides the person being treated. For example, the use of antibiotics may increase antibiotic resistance, and not being vaccinated for COVID-19 or not avoiding contact with other people may increase the risk of infection for others. Similarly, when decisions are made for a group of people rather than for individuals, the outcomes that are important to anyone who is affected should be considered.

Consider the relevance of fair comparisons in laboratories, animals or highly selected people

Studies that only include animals, or only a selected minority of people, may not provide results that are relevant to most people.

Table 1 shows some examples of misleading extrapolation from animals or a selected minority of people, found in news reports. ³

Table 1.

Examples of misleading extrapolation from animals or a selected minority of people found in news reports.^a

Quote from news reports	Basis for the quote
‘Researchers have shown that contact lenses laced with medicines are an effective way of treating glaucoma patients.’	A study that showed the effect only in rabbit eyes.
‘It could treat phobias and perhaps even post-traumatic stress disorders.’	A before–after study in 15 healthy volunteers without any phobia.
‘Broccoli slows arthritis’.	A study in mice of a sulphoraphane compound present in cruciferous vegetables, including broccoli.
‘The results of the trial – the first in humans – could offer hope to one in five people who are resistant to statins. It could also be offered to patients who suffer ill-effects from the drugs, or those whose cholesterol remains high even after statins are prescribed.’	A study in healthy volunteers with high cholesterol levels who had received no lipid-lowering treatment in the past 30 days and were not statin resistant.
‘Everyone should have at least 10–15 minutes of exposure to the sun every day to ensure that vitamin D levels are adequate.’	A study in rats that assessed dietary vitamin D deficiency leading to elevated tyrosine nitration in the brain, which may promote cognitive decline.

^aBased on Haneef et al. ³

Consider the relevance of the treatments that were compared

A fair comparison of the effects of a surgical procedure done in a specialised hospital or delivered by an experienced surgeon may not provide a reliable estimate of its effects and safety in other settings, or in the hands of less experienced surgeons.

For example, results from a large randomised trial showed that endarterectomy (surgical removal of part of the inner lining of an artery) for asymptomatic carotid stenosis (narrowing of the large arteries on each side of the neck) reduced the five-year absolute risk of stroke by about 5%. ⁴ However, the trial only accepted surgeons with a good safety record, rejecting 40% of applicants and subsequently barring those who had adverse operative outcomes in the trial from further participation. The benefit from surgery was largely attributable to the low operative risk. Operative mortality was eightfold higher outside of the trial and the risk of stroke and death was about threefold higher.

Similarly, comparing a new drug to a drug or dose that is not commonly used (and which may be less effective or safe than those in common use) would not provide a relevant estimate of how the new drug compares to what is commonly done.

For example, in randomised trials of atypical antipsychotics for schizophrenia, haloperidol (one of the most frequently prescribed ‘typical’ antipsychotics worldwide) was used as the comparison treatment. ⁵ However, the trials used haloperidol in doses that were higher than that recommended. In a meta-analysis of 52 randomised trials that controlled for the higher-than-recommended dose of comparator drugs, differences in effectiveness and overall tolerability between typical and atypical antipsychotics disappeared, suggesting that the perceived benefits of atypical antipsychotics were due to excessive doses of the comparison treatments, such as haloperidol. ⁶

Consider the relevance of the circumstances in which the treatments were compared

Some treatment comparisons are designed to find out if a treatment can work under ideal circumstances, for example, with people who are most likely to benefit and most likely to comply, and with highly trained practitioners who deliver the treatment exactly as intended. These comparisons, which are sometimes called ‘explanatory’ or ‘efficacy’ studies, may not reflect what happens under usual circumstances.

The North American Symptomatic Carotid Endarterectomy Trial is an example of an ‘explanatory randomised trial’.^7–9 It demonstrated that a lower risk of stroke was experienced by highly selected patients with severe narrowing of the large arteries on either side of the neck (carotid stenosis). Those patients who had a high risk of stroke were most likely to respond to surgical removal of the inner layer of those arteries (endarterectomy). Participating patients were closely followed and were operated on by skilled surgeons in academic and specialist hospitals with adherence to a strict protocol. The trial showed that endarterectomy reduced the risk of stroke under those ideal circumstances, but it did not provide a reliable estimate of the beneficial and harmful effects of endarterectomy under more typical circumstances.

Implications

• Make sure you are considering the correct diagnosis or problem, and appropriate options for addressing it.

• Always consider the possibility that important outcomes may not have been addressed in fair comparisons. Avoid being misled by surrogate outcomes.

• Results of systematic reviews of studies in animals, or highly selected groups of people, may be misleading.

• Be aware that treatments available to you may be sufficiently different from those in the available research studies that the results of those studies may not apply to you.

• Be aware that the results of studies with the aim of finding out if a treatment can work may overestimate the benefits of a treatment given under more usual circumstances.

Declarations

Competing Interests

None declared.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Research Council of Norway (Project numbers 220603/H10 and 284683). The funder had no role in the decision to publish, or preparation of the manuscript.

Ethics approval

Not applicable.

Guarantor

ADO.

Contributorship

ADO, IC and AD conceptualised, reviewed and edited drafts of this essay. ADO prepared the first draft.

Provenance

Not commissioned; invited article from the James Lind Library.

Note

Additional material for this article is available from the James Lind Library website [www.jameslindlibrary.org], where it was previously published.