Skip to main content
NCBI home page
F1000Research logoLink to F1000Research
. 2016 Apr 18;5:693. [Version 1] doi: 10.12688/f1000research.8392.1

Better Outcomes through Learning, Data, Engagement, and Research (BOLDER) – a system for improving evidence and clinical practice in low and middle income countries

BOLDER Research Group
PMCID: PMC5089159  PMID: 27853508

Abstract

Despite the many thousands of research studies published every year, evidence for making clinical decisions is often lacking. The main problem is that the evidence available is generated in conditions very different from those that prevail in routine clinical practice and with patients who are different. This is particularly a problem for low and middle income countries as most evidence is generated in high income countries.

A group of clinicians, researchers, and policy makers met at Bellagio in Italy to consider how more relevant evidence might be generated. One answer is to conduct more pragmatic trials—those undertaken in routine clinical practice. The group thought that this might best be achieved by developing “learning health systems” in low and middle income countries.

Learning health systems develop in communities that include clinicians, patients, researchers, improvement specialists, information technology specialists, managers, and policy makers and have a governance system that gives a voice to all those in the community. The systems focus on improving outcomes for patients, use a common dataset, and promote quality improvement and pragmatic research. Plans have been developed to create at least two learning systems in Africa.

Keywords: BOLDER, clinical decision making, learning health systems, low income countries, middle income countries, Africa

Between 2% and 53% (median 19%) of treatments offered to patients lack substantial research to support them 1. A study of 16 guidelines from the American College of Cardiology and the American Heart Association found that only 314 (11%) recommendations of 2711 were supported by the highest level of evidence 2; and cardiovascular medicine is probably the best researched part of clinical practice.

This deficiency is even more serious in low and middle income countries because most research is conducted in high income countries and may not be applicable in low and middle income countries. Those countries have rapidly rising rates of non-communicable disease (NCD), but an analysis of the 633 systematic reviews related to NCD found that almost 90% of 8850 trials included in the reviews were from high income countries, 5% from low-middle income countries, and only 13 (0.15%) from low income countries 3.

At the same time as we have inadequate evidence to support many clinical decisions, clinicians are wary when we do have evidence, doubting its relevance to their local situations. In combination, these two kinds of evidence deficiency are depriving patients of access to the best treatments. How might more useful evidence be produced more efficiently? A group of clinicians, researchers, and policy makers, mostly from Africa, met in 2015 at the Bellagio Centre and developed some tentative answers.

What is useful evidence?

Useful evidence has two components. It must be internally valid in that users of the research can be confident that its conclusions are supported by its methods and results. But it must also be externally valid, meaning that it is applicable in a wide range of circumstances, in the "real world" as opposed to the ideal world common in most clinical trials. This problem of the "applicability" of research is particularly acute for those in low and middle income countries as most research has been conducted in high income countries 3. Lower income countries need evidence on their own health care challenges, and they need it to be generated within their populations by their patients, clinicians, and researchers.

Studies, particularly clinical trials, may lack internal validity because they are too small, too short term, fail to remove bias, too poorly done, use surrogate outcome measures irrelevant to patients and unconvincing to clinicians, or too poorly reported. A study of 2000 randomised trials in schizophrenia found that most were not useful for making clinical decisions: studies were short (54% lasted less than six weeks), small (mean number of patients 65), and poorly reported (64% had a quality score of less than or equal to two when the maximum score was five) 4. Furthermore, the studies tested over 600 different interventions and used 640 different rating scales to measure outcomes, making interpretation for clinical use almost impossible 4.

External validity may be lacking because the patients are highly selected, excluding, for example, the old and those with multiple conditions, the research setting is not like those in which the treatment will be applied, the conditions of the research protocol are highly controlled, and patients monitored in a way that is not possible in everyday practice. Most drug trials fall into this category because they are what is required by regulators to allow drugs into the market. Furthermore, the drugs may be tested against placebo, when the question that matters to clinicians and policy makers is whether they are better than other currently used existing treatments, not only other drugs. Applicability is a particular problem in paediatrics as most studies are conducted in adults.

More relevant research

So why not make research more relevant and - at the same time - more effective? At Bellagio our working group developed a concept called BOLDER (Better Outcomes through Learning Data and Engaging in Research: www.bolderresearch.org). One key element of BOLDER is pragmatic research. Pragmatic studies are those conducted in routine clinical practice settings, and patients are enrolled with few selection criteria in order to maintain the representativeness of the true population 5. In addition, the organisation of the studies should be simple, as few extra data as possible should need to be collected, and the outcome measures used should matter to those who take part in the trial, both patients and clinicians 6, 7. The hope is that the clinicians or policy makers will accept the results of the trial and act on them. Ideally these studies should be conducted rapidly and cheaply, avoiding the long delays and substantial costs of many trials, and be largely done by the clinicians who are the main consumers of clinical research.

Learning health systems

Before the meeting it wasn't clear how this more useful research might be achieved, particularly in Subsaharan Africa, but during the meeting a possible answer emerged--the creation of “learning health systems.” A learning health system is one in which patients and providers work together to coproduce new knowledge and share decisions regarding best evidence 8. It drives discovery but is a natural outgrowth of patient care. It increases innovation, quality, and safety, and does this in real time.

Quality improvement science identifies barriers to improving health outcomes, finds ways to try and overcome them, evaluates the impact of interventions, and - if services and patients’ outcomes improve - keeps the cycle of improvement going. But the worlds of quality improvement and formal “research” rarely collide. Systems that bring these two worlds together do now exist, however, in a few places in the US and Europe. At the meeting we heard about ImproveCareNow 9, which began in 2007 when eight paediatric gastroenterology practices came together to improve the care of children with inflammatory bowel disease. Agreeing on an outcome measure of remission, the system established a common dataset, standardised care, and engaged patients and families. Using cycles of improvement it increased remission rates over seven years from an average of 50% to 80%. During that time it grew from eight practices to over 80 9. The system then began to conduct research studies, thus becoming a true learning health system.

ImproveCareNow served as the prototype for a national, multispecialty learning health system called PEDSnet, which is now expanding to include many more hospitals and children and is conducting several pragmatic trials 10. It is part of a wider network, PCORnet, that includes 12 other networks like PEDSnet. PCORnet provides care to 75 million Americans and is an unsung benefit of Obamacare.

The six components of a learning system

A successful learning system has six components.

  • A community, which ideally will include clinicians, patients, researchers, improvement specialists, information technology specialists, managers, and policy makers.

  • A focus on outcomes. The learning health system must produce better outcomes for patients. If it doesn't it will --and should--fold.

  • A common dataset that is as simple as possible with data being entered only once. Extra data might be collected for particular studies.

  • Quality improvement, which reliably applies evidence generated from research when and where patients can benefit.

  • Pragmatic research

  • Governance, which should ensure a voice for all those in the system, particularly patients.

A learning system for Africa?

But could a learning health system work in Africa? The conviction of those at the meeting was that it could. Nascent platforms were identified in Kenya, Malawi, Zambia, and South Africa, and interrogation of leaders from the Kenyan and South African platforms made those at the meeting think that learning health systems could be developed in those two countries at least 11.

In Kenya the Wellcome KEMRI network of hospital paediatricians has developed a core dataset that is collected on every single patient who is admitted and is able to conduct research using these data. Several important randomised controlled trials have already been completed using this platform 12, 13.

In South Africa there is a well developed national system to incorporate current evidence based clinical guidelines into daily clinical practice in primary care. The guidelines reach tens of thousands of nurses and doctors and have improved the care of millions of patients across the entire country 14.

In BOLDER we are working to build on these capacities. The aim in Kenya is to develop a learning health system that can rapidly implement into daily practice across the country the evidence it gathers from pragmatic research. In South Africa we hope to build a basic electronic data platform that can be used in routine care in even the most remote facilities but can also be used to conduct research in these real world practice settings.

Conclusion

A potential answer to the problem of inadequate evidence for clinical practice, particularly in Africa, has become clear. A learning health system will be built on networks that have already been scaled up in two countries in Africa. The systems will concentrate on improving outcomes and include all stakeholders, including patients, clinicians, and researchers. The research will happen in a context that allows it to be quickly implemented, and the aim is for the research to be pragmatic and be done quickly and cheaply. Plans are launched to make it happen.

Note

The first draft of this article was written by RS, and there is some overlap with a blog he posted immediately after the meeting: http://blogs.bmj.com/bmj/2015/08/11/richard-smith-how-to-fill-the-void-of-evidence-for-everyday-practice/

Funding Statement

The stay at Bellagio was funded by the Rockefeller Foundation.

[version 1; referees: 2 approved]

References

  • 1. Jamtvedt G, Klemp M, Mørland B, et al. : Responsibility and accountability for well informed health-care decisions: a global challenge. Lancet. 2015;386(9995):826–828. 10.1016/S0140-6736(15)60855-8 [DOI] [PubMed] [Google Scholar]
  • 2. Tricoci P, Allen JM, Kramer JM, et al. : Scientific evidence underlying the ACC/AHA clinical practice guidelines. JAMA. 2009;301(8):831–841. 10.1001/jama.2009.205 [DOI] [PubMed] [Google Scholar]
  • 3. Heneghan C, Blacklock C, Perera R, et al. : Evidence for non-communicable diseases: analysis of Cochrane reviews and randomised trials by World Bank classification. BMJ Open. 2013;3(7): pii: e003298. 10.1136/bmjopen-2013-003298 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4. Thornley B, Adams C: Content and quality of 2000 controlled trials in schizophrenia over 50 years. BMJ. 1998;317(7167):1181–4. 10.1136/bmj.317.7167.1181 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5. Raymond J, Darsaut TE, Altman DG: Pragmatic trials can be designed as optimal medical care: principles and methods of care trials. J Clin Epidemiol. 2014;67(10):1150–6. 10.1016/j.jclinepi.2014.04.010 [DOI] [PubMed] [Google Scholar]
  • 6. Tosh G, Soares-Weiser K, Adams CE: Pragmatic vs explanatory trials: the pragmascope tool to help measure differences in protocols of mental health randomized controlled trials. Dialogues Clin Neurosci. 2011;13(2):209–15. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7. Loudon K, Treweek S, Sullivan F, et al. : The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147. 10.1136/bmj.h2147 [DOI] [PubMed] [Google Scholar]
  • 8. Institute of Medicine (US) Roundtable on Evidence-Based Medicine; . Olsen LA, Aisner D, et al. : The Learning Healthcare System: Workshop Summary. Washington (DC): National Academies Press;2007. 10.13184/eidon.39.2013.89-91 [DOI] [PubMed] [Google Scholar]
  • 9. Greene SM, Reid RJ, Larson EB: Implementing the learning health system: from concept to action. Ann Intern Med. 2012;157(3):207–10. 10.7326/0003-4819-157-3-201208070-00012 [DOI] [PubMed] [Google Scholar]
  • 10. Forrest CB, Margolis P, Seid M, et al. : PEDSnet: how a prototype pediatric learning health system is being expanded into a national network. Health Aff (Millwood). 2014;33(7):1171–7. 10.1377/hlthaff.2014.0127 [DOI] [PubMed] [Google Scholar]
  • 11. English M, et al. : Building Learning Networks to accelerate research and uptake of evidence to improve quality and outcomes of clinical care in low-income settings. Submitted for Publication. [Google Scholar]
  • 12. English M, Mohammed S, Ross A, et al. : A randomised, controlled trial of once daily and multi-dose daily gentamicin in young Kenyan infants. Arch Dis Child. 2004;89(7):665–669. 10.1136/adc.2003.032284 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13. Agweyu A, Gathara D, Oliwa J, et al. : Oral amoxicillin versus benzyl penicillin for severe pneumonia among Kenyan children: a pragmatic randomized controlled noninferiority trial. Clin Infect Dis. 2015;60(8):1216–24. 10.1093/cid/ciu1166 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14. Fairall L, Bateman E, Cornick R, et al. : Innovating to improve primary care in less developed countries: towards a global model. BMJ Innov. 2015;1(4):196–203. 10.1136/bmjinnov-2015-000045 [DOI] [PMC free article] [PubMed] [Google Scholar]
F1000Res. 2016 Oct 31. doi: 10.5256/f1000research.9025.r17309

Referee response for version 1

Lilian Dudley 1

Overview

This opinion paper addresses important concerns about the gap between clinical evidence generation, particularly clinical trials in high income countries (HIC), their appropriateness for low and middle income country (LMIC) health systems, and the challenges of translating research evidence into practice in different settings.

The authors propose the use of pragmatic research (trials) methods in LMIC’s within the framework of ‘learning health systems’ (LHS) as a potential solution to the current problems. The proposed approach suggests ways of conducting clinical research within ‘real world’ settings, and mechanisms by which the research evidence can be more appropriate, and facilitate implementation by practitioners and patients particularly in LMIC’s. The opinion paper generates helpful ideas and discussion on ways of reducing the disconnect between knowledge of what works in selected or ideal settings, and producing more appropriate local knowledge with the involvement of practitioners as the primary users of the knowledge.

Frameworks of LHS, HSR and QI

The authors have applied components of LHS’s to the use of clinical research evidence in LMIC’s. The interpretation of LHS has not fully explained the overall concept and framework for LHS, and its application has only used a limited perspective of LHS’s in providing a framework for clinical research methods in LMIC’s.

LHS’s seek to address concerns re external validity and use of evidence collected through trials, and is therefore an appropriate model to consider for ‘knowledge translation’. However, what the authors do not elaborate is that it does so largely through using ‘evidence’ from electronic patient information systems, and by promoting access to, interoperability and use of electronic patient records (Big Data) to provide more patient centered evidence. This ensures that evidence is linked to real patients in practice, and can be used by real health providers and patients. It benefits from the growth of ICT’s in health in HIC’s, and the extensive access to and improved quality of routine patient information in such settings. This is underpinned by financial and human resources, and infrastructure which is able to support such ICT’s for health in HIC’s.

The authors appropriately discuss the re-conceptualisation of ways of doing research i.e. shifting from a pre-occupation with internal validity to ensuring external validity; moving from the ‘ideal’ to ‘pragmatic’ approaches; and collaborations across ‘institutions’, between researchers and practitioners and patients.  A component of LHS which could further enhance this reconceptualization is the focus on ‘harvesting’ existing data on real patients from RHIS, rather than ‘hunting’ data on ideal selected group of patients through the expensive and time consuming collection of new data with limited external validity. One of the challenges of such an approach however is that ‘Big data’ and electronic patient record systems do not exist in many LMIC’s. Many have very basic routine health information systems (RHIS), in which the quality of the data is often poor; and very few have electronic patient record systems to generate the kind of evidence needed for clinical decision making and to support a new way of doing research.

Although the opinion piece speaks to LHS, by focusing narrowly on the potential of ‘pragmatic’ trials as a research approach it does not address fully how LHS can be used to promote the generation of evidence from different sources and its translation into practice. It would be useful to explore further the opportunities which LHS present to review the nature of evidence, and the kind of research which can be conducted i.e. sources of data, greater patient centredness, and the role of clinicians/practitioners as researchers and users of evidence in LMIC settings.

By limiting the definition and understanding of LHS to one which simply conducts pragmatic trials rather than RCT’s, it misses the opportunity for researchers to rethink the model of research in LMIC’s, and the links between research, RHIS, and the opportunities it can create for ‘co-creation’ and use of knowledge between researchers, practitioners, patients and families to inform and transform practice. It would be useful if the authors could comment on these issues related to LHS and its application in LMIC’s, and whether any consideration was given to it in the Bellagio meeting.

The proposal also draws on different approaches to research already practiced as part of ‘health services’ or ‘health systems’ research. The authors propose more collaborative and participative research practices, without referring to the broad and well established field of participatory action research (PAR), which facilitates ‘transformative’ research. PAR specifically aims to include users of knowledge in the research process in order to empower and build capacity of users of research (and researchers) to ensure implementation and sustainability of interventions. LHS and its application in this opinion paper, is therefore not new in identifying processes for greater involvement of users, and it would be helpful to reflect on how lessons from PAR for supporting ‘action learning’ can support such a process.

The proposal also draws on Quality Improvement, through which evidence is implemented in collaborative and iterative processes to improve practice, and through which both researchers and practitioners learn.  The artificial divide between QI and research has diminished substantially in recent years and QI is increasingly underpinned and supported by QI research on a continuum between research and practice. The HIC projects which the authors identify are useful examples of QI supported by research. It would have been helpful if the authors could identify similar QI projects in LMIC where research is a core component of the project to illustrate that this is indeed feasible in such settings.

Pragmatic Trials as an approach

In resource constrained environments such as in LMIC’s, barriers to doing RCT’s include limited financial resources, research and institutional capacity to support any form of trials. The paper does not indicate how pragmatic trials will overcome these constraints to make the kinds of evidence more easily producible and accessible in LMIC’s.

Page 2, para 5: Although the emphasis of LHS is not on improving internal validity, the authors do raise some concerns about internal validity of clinical trials. However, the article then does not adequately address how pragmatic trials as part of LHS’s can overcome these internal validity shortcomings of RCT’s. If LHS and pragmatic trials are not addressing internal validity, perhaps this section should not receive the level of emphasis – or it needs to be addressed in the context of LHS and pragmatic trials.

Page 2, para 7: ‘as few extra data as possible should need to be collected’ – If the LHS approach to pragmatic trials includes increased use of routine data, it is important to discuss whether suitable data is available, and indicate how pragmatic trials will overcome the limitations of routine data in many LMIC’s, as well as the lack of trust in the data by clinicians.

Page 2, para 7: The emphasis on clinicians doing the studies may not fully take the reality of LMICs into account. Clinicians are a scarce resource in most LMIC’s and tend to be overburdened with clinical work with little time to undertake additional research activities. Many also do not have the necessary training and research skills to participate in such studies. It would therefore be useful for the opinion piece to indicate how and why it will be easier under these circumstances for clinicians to do pragmatic trials as opposed to other forms of trials or research; and how LHS will contribute to facilitating this and building capacity to undertake these trials.

Page 2, para 8, proposes a more collaborative and participative research process, but does not reference the extensive experience and lessons of PAR – which precisely aims to not only undertake research under real world conditions, and involve participants and stakeholders and create a learning process for both. Much could be learnt by pragmatic triallists by considering PAR approaches which already describe ways of engaging and joint learning, but also highlight some of the limitations.

Page 2, para 9: There is a continuum from QI practice to QI research which is increasingly recognised and practiced. Unfortunately the only examples provided of QI research are from HIC projects, and it would have been useful to identify examples of QI research in LMIC’s. It does not help the authors’ argument that LHS need to be local in LMIC, if there is little evidence of that happening.

Page 3, para 2: Patient centredness is a key element of LHS which receives little focus in the paper. In HICs’ an important component of using electronic patient data is that the data is linked directly to real patients, and can include their experiences and assessments of outcomes. It would be useful for the opinion paper to indicate how this aspect can be used and applied in LMIC’s as part of pragmatic research.

Page 3, para 4 – the database of clinical guidelines in SA, although a step in the right direction, does not really encompass the scope of activities proposed in the paper or typical of a LHS. It would be helpful if the authors could indicate how this links to the earlier suggestions of pragmatic trials, and the involvement of clinicians in research.

Page 3, para 5: The suggestion that new electronic data collection systems for research be established is also contrary to earlier suggestions of limiting the requirement to collect new data. There have been several initiatives to establish electronic patient records in South Africa and the authors should rather investigate what has already been developed and implemented.  LMIC’s are littered with projects which have set up independent data collection systems instead of strengthening the routine information systems, and this needs to be approached with caution.

Page 3, para 6: Before proposing this as  a ‘clear’ solution, it would be useful to have more engagement with clinicians, policy makers, patients, ICT personnel and to look at current practices in LMIC’s in order to develop a clearer framework for how LHS’s and pragmatic research can support the use of evidence in LMIC’s.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

F1000Res. 2016 Jul 27. doi: 10.5256/f1000research.9025.r14218

Referee response for version 1

Gillian Mann 1

Please note that I am writing in my personal capacity; this is not an official response from the Department for International Development

This is a helpful contribution to the health research agenda. For some time researchers have been asking how to have research taken into policy - this goes some way to answer questions of why that does not often happen. 

It would help if more examples outside the specific areas of clinical practice were given. For example research may not only be on one narrow interest topic (though this clearly has validity), but may be on broader approaches to patient care - e.g. improving outreach services in community based care, reaching neglected patient groups with services that already work for others, changing broad clinical protocols to improve patient pathways to care.

It would also help to highlight some of the contextual reasons why research has failed to be taken up in the past - e.g. cost of new technologies (not just cost effectiveness), a lack of understanding of what is needed to make new technologies work (e.g. staffing skills, numbers, distribution; consistent availability of water or electricity etc). Previous work relating to the types of evidence required has previously been undertaken 1.

Two phrases are used rather interchangeably throughout: "Learning Health Systems" and "Learning Systems".  I would argue that the latter is more applicable in this context.  A "learning health system" implies that the broad health system is learning, whereas this article refers to establishing systems to learn to improve clinical practice in particular, which may only relate to parts of the health system, circumscribed by geography, field or research, or some other criteria. While this may evolve into a broader learning health system, the term is rather grander than the scope at this time.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

References

  • 1. Mann G, Squire SB, Bissell K, Eliseev P, Du Toit E, Hesseling A, Nicol M, Detjen A, Kritski A: Beyond accuracy: creating a comprehensive evidence base for TB diagnostic tools. Int J Tuberc Lung Dis.2010;14(12) :1518-24 [PubMed] [Google Scholar]

Articles from F1000Research are provided here courtesy of F1000 Research Ltd

RESOURCES