Rivers of Evidence

Abstract

There has been too much of a one-way flow drift down a river of evidence. Researchers from rich countries have produced the primary evidence which they proceed to summarise within reviews. These summaries have directed care worldwide. However, things are changing and the river of evidence can flow in the other direction. The care of women with eclampsia has been changed or refined throughout the world because of a large low and middle income country trial. The global care of people with heart disease has been greatly modified by studies originating in China. The care of people who are acutely aggressive because of psychosis has to be reconsidered in the light of the evidence coming from Brazil and India. Healthcare is an issue everywhere and evaluation of care is not the premise of any one culture—the evidence—river must run both ways.

Background

If possible, before a treatment is widely used, the intervention should be fairly tested and all best evidence summarised. Too often, this type of research has been undertaken in a few wealthy countries with a tradition of evaluation and synthesis of evidence. This does not have to continue.

i. Randomized trials

Randomised controlled trials, the human experiment, if possible and ethical, remain the best source of the effects of any treatment (1). Not only will all issues known to affect outcome be evenly distributed by randomisation (for example issues such as age, sex, and duration of illness), but also variables not yet fully understood will be equal across groups of allocation (for example genetic pre-disposition or differing lifestyle). There are many limitations to the randomised trial methodology. For example, when it comes to identification of rare but important adverse effects, relatively small trials of short duration may fail to find such outcomes (a design limitation). Furthermore, reporting of adverse effects may be problematic or biased—a limitation of conduct—that can reduce the value of any specific randomised trial. Nevertheless, despite limitations or threats to validity, the well planned conducted and reported randomised controlled trial remains the ‘gold standard’ by which care is evaluated.

ii. Reviews

No one can know all relevant controlled trials in their area. The need to summarise the totality of evidence has been long-recognised—even to do that in a way that is systematic and scientific (2)—but, reviews of the literature have traditionally been qualitative, based on experience, wisdom and consensus. Only relevantly recently has this process of reviewing been the wide focus of careful scientific scrutiny in itself. Pioneers in this ‘secondary research’ piloted systematic review techniques (3) that have now evolved, and became partly industrialised in the form of the Cochrane Collaboration (4). This Collaboration facilitates the production and maintenance of systematically conducted reviews of the effects of healthcare. It supplies training, general support, and free software for these reviews to be conducted, and a means by which they are globally distributed (within the Cochrane Library).

Direction of flow

The rivers of evidence, however, have tended to flow in one direction. Researchers from rich countries have tended to undertake both the trials and the underpinning systematic reviews, making interpretation and applicability of results to less affluent healthcare systems problematic. Care of people with schizophrenia is an interesting and useful example. This illness has a lifetime prevalence of 1%, irrespective of social class, affluence, region or race (5). We therefore know that 1% of people with schizophrenia will live in Iran, another 1% in the UK, 5% in the USA, 20% in India and 80% in the group of low or middle-income countries. Sheer weight of population predicts societal burden of illness. However, the best predictor of national trial output for schizophrenia randomised trials is Gross Domestic Product (GDP)—not the number of people with schizophrenia in that country, nor the estimate of the spread of wealth per capita, nor indicators of technological advance (6). The USA, with only 5% of the world’s population of people with the illness, produces more than 50% of all relevant trials. The USA, Europe and Australia are generous producers of randomised studies, the results of which are used worldwide. Largely these trials, understandably, focus on people within the ‘home’ population and care culture. For healthcare professionals, as well as patients in the Middle and Far East, in India as well as across South America, there is the issue of giving or receiving treatment because of findings from trials designed in very different settings. These trials, then summarised by consensus, were disseminated within reviews that tended to flow out from the producer nations—US—generated data tended to be reviewed by US reviewers and then disseminated worldwide. This situation was further perpetuated by migration of well-trained doctors from less affluent care cultures to high-income countries. Upon return, these doctors would have influenced and helped encourage adoption of the affluent-society patterns of care that they have become familiar with.

There are really good things within this flow on the river of evidence. Although the methods of fairly testing interventions have been known for hundreds of years (7), the first widely recognised randomised controlled trial came out of post-World War II England (8). The first systematic review, although preceded by an understanding of what was needed by centuries, was published by the US (3,9). Methods have substantially evolved and now a global accessibility to methods of undertaking systematic reviews through the Cochrane Collaboration started in England (10).

Changes in direction

The future is likely to bring many more systematically conducted reviews, mirroring greater numbers of randomised controlled trials. Healthcare professionals in poor countries have always generated traditional reviews, qualitative rather than quantitative, based more on experience and wisdom. The systematic and quantitative reviews of care tended to come from more privileged care and research traditions. Although the traditional and the systematic review may concur in their findings, sometimes they do not, and it is difficult to appreciate where discrepancy is occurring—authoritative traditional reviews have shown to be recommending lethal interventions (11). The rich-down direction of systematic evidence flow need not necessarily continue. The logic of systematic reviews is now widely acceptable, training is available, tools are universal and disseminations are often free. For example, as the war in Syria continues, researchers support the insatiable curiosity of groups of medical students by running workshops on systematic reviews in Damascus (12). As a result, the Cochrane Schizophrenia Group has more systematic reviews coming from Damascus than it does from the whole of the USA. These reviews, not only are applicable to the people of Syria, but are used worldwide and have been incorporated into WHO guidance (13).

These reviews, however, tend to summarise trials undertaken by privileged societies. Out of necessity, the expensive highly explanatory trials have been undertaken in rich countries. Regulatory authorities demand high levels of certainty regarding the diagnosis of who enters a trial, rather rigid care regimes, and scrupulous measurement of outcomes (14). The nature of this design, of course, makes it problematic to interpret for every day care where people may not be as thoroughly diagnosed, care regimes do vary from day to day, and outcomes are not measured on rating scales but are to do with life functioning, service utility and obvious side effects. Only relatively latterly have clinicians and consumers opinion been heeded and more pragmatic design introduced. This latter type of study tends to be more clinically valuable (14)—but also less expensive to undertake.

There are examples right across healthcare. The care of women with eclampsia has been changed or refined throughout the world because of a large trial coming from low and middle income countries (15). The global care of people with heart disease has been greatly modified by studies originating in China (16). There are good examples from mental health care too. The series of TREC (Tranquilização Rápida-Ensaio Clínico), translated as rapid tranquillisation-clinical trials, from Brazil and India were designed by researchers in these countries who had discovered that treatments for people were acutely violent because psychotic illnesses were grossly under-researched. The teams undertook the world’s largest studies in this area, and recorded outcomes that were of interest to these care cultures (17–21). These trials have been widely recognised as being “unlike most of the other studies in this [area of research, the TREC studies], were large studies of a high methodological quality” (22). These randomised trial studies did not copy what had been done before—they bettered those methods. All ran on infrastructure funding alone. They were undertaken by interested researchers and clinicians, because they wished to know the answer. These researchers wanted to disseminate good evidence to the rest of the world, and show the research community that good evidence can come from less privileged situations. They wanted to show the perverse advantage of those less privileged situations for generation of good, very widely applicable, evidence. As researchers in less privileged situations gain confidence, are supported, look out into the world of research and have access to disseminate in globally accessible forms, more two directional flow of the river can be expected.

Conclusion

With courage and leadership, good science is possible almost anywhere. If allowed to question, and doors are not held closed, interested researchers can objectively evaluate treatments employing ‘gold standard’ methodology—with enormous repercussions on local and international care. Although there are examples of pioneering work from the Middle East, these examples remain too few. There are the same numbers of people with, for example, schizophrenia, in the Middle East as there are in the USA, yet only a handful of trials come from Arab or Persian lands (23) and this is not all to do with GDP. This dearth of local good evidence applies to other areas of medicine too. For far too long, the river has flowed only one way. Generous political and research leadership would allow a natural flourishing of local research—for, at the very least, local needs. The global accessibility of systematic review techniques and recognition of the importance of pragmatic randomised trials are opportunities for anyone wanting to generate high grade evidence of better quality than has been seen before.

Ethical issues

Not applicable.

Competing interests

The author declares that he has no competing interests.

Author’s contribution

CEA is the single author of the manuscript.

Citation: Adams CE. Rivers of evidence. International Journal of Health Policy and Management 2013; 1: 247–249.