From the EBM pyramid to the Greek temple: a new conceptual approach to Guidelines as implementation tools in mental health

Abstract

Guideline methods to develop recommendations dedicate most effort around organising discovery and corroboration knowledge following the evidence-based medicine (EBM) framework. Guidelines typically use a single dimension of information, and generally discard contextual evidence and formal expert knowledge and consumer's experiences in the process. In recognition of the limitations of guidelines in complex cases, complex interventions and systems research, there has been significant effort to develop new tools, guides, resources and structures to use alongside EBM methods of guideline development. In addition to these advances, a new framework based on the philosophy of science is required. Guidelines should be defined as implementation decision support tools for improving the decision-making process in real-world practice and not only as a procedure to optimise the knowledge base of scientific discovery and corroboration. A shift from the model of the EBM pyramid of corroboration of evidence to the use of broader multi-domain perspective graphically depicted as ‘Greek temple’ could be considered. This model takes into account the different stages of scientific knowledge (discovery, corroboration and implementation), the sources of knowledge relevant to guideline development (experimental, observational, contextual, expert-based and experiential); their underlying inference mechanisms (deduction, induction, abduction, means-end inferences) and a more precise definition of evidence and related terms. The applicability of this broader approach is presented for the development of the Canadian Consensus Guidelines for the Primary Care of People with Developmental Disabilities.

Introduction

Guidelines are implementation tools which evolved within the framework of evidence-based medicine (EBM). The EBM structures used to develop a guideline include the formulation of a clinical question (population, intervention, comparison, outcome commonly referred to as PICO) (Guyatt et al. 2008; World Health Organization WHO, 2012); standards and a protocol for systematic searching of empirical research evidence on this question; and quality appraisal of the research followed by use of an analytic framework to formulate and grade the recommendations from the body of research evidence. The principle underpinning guidelines is to guide health care practice and optimise patient care by reducing practice variability; improve quality and maximise resource utilisation (World Health Organization WHO, 2012; National Institute for Health and Care Excellence, 2016), the premise being that quality peer reviewed published scientific research is the best evidence of effectiveness (what interventions work and which do not work in real practice).

Over the past 25 years, guideline development has become standard practice. In August 2016, the Library on the Guideline International Network (GIN) website listed 48 guidelines on depression, 35 on schizophrenia and 47 on anxiety in different languages (Guidelines International Network GIN, 2016). Yet the translation of research evidence, implementation of guideline recommendations to health care practice faces significant challenges (Grimshaw et al. 2004; Grol & Buchan, 2006), particularly in highly complex areas such as mental health, where guidelines should be developed under conditions of high ambiguity, uncertainty and variability (Salvador-Carulla et al. 2013). A recent systematic review of studies on the implementation of mental health guidelines concluded that guideline implementation does not appear to have an impact on provider performance, although it may influence patient outcomes (Girlanda et al. 2016).

What are the mainstream solutions to improve guideline development?

One of the earliest concerns was the disparate methods and procedures used to develop guidelines and their recommendations in the 1990–2000s. These concerns led to the development of standards for guideline development (Institute of Medicine, 2011; National Health and Medical Research Council, 2011; Qaseem et al. 2012; National Institute for Health and Care Excellence, 2016); the AGREE guideline appraisal tool (Brouwers et al. 2010); and grading systems and tools such as GRADE (Guyatt et al. 2008; Goldet & Howick, 2013).

Aligned to this methodological problem were the inconsistencies in the strength applied to recommendations where the same research evidence was used (Guyatt et al. 2008); the cost of duplication of guidelines and the subsequent appeals to adapt guidelines to different contexts rather than develop guidelines de novo (Fervers et al. 2006; The ADAPTE Collaboration, 2009). Other concerns referred to the trustworthiness of guidelines due to potential conflicts of interest of experts involved in their development (Minhas, 2007; Mitchell et al. 2016; Shnier et al. 2016) which led to new standards for reporting and managing financial and intellectual conflicts (Boyd & Bero, 2006; Guyatt, 2010).

The overall usability and applicability of guideline recommendations to different contexts and for different targets has also raised concerns specifically around the limited attention to variability in the real world and to local environmental factors. In addition, there are difficulties in accommodating person-centred health care principles of shared decision making, patient values, preferences, choice and control to the recommendations (Charles et al. 2011; Rosner, 2012; Salvador-Carulla et al. 2013; Greenhalgh et al. 2014, 2015; Fernandez et al. 2015; Bingeman, 2016). A series of biases related to this challenge have been enumerated (Charles et al. 2011; Rosner, 2012; Salvador-Carulla et al. 2013; Greenhalgh et al. 2014, 2015; Fernandez et al. 2015; Bingeman, 2016). Increasing emphasis has been placed on the involvement of consumers in the development of guidelines (Schunemann et al. 2006; Nilsen et al. 2010; Guideline International Network (GIN), 2015). Other authors have called for improving implementation strategies, for developing new tools and frameworks to support use of guidelines in the real world (Shiffman et al. 2005; Doherty, 2006; Francke et al. 2008; Hakkennes & Dodd, 2008; Gagliardi et al. 2011; Brouwers et al. 2015; Fleuren et al. 2015); and to incorporate the analysis of the context and the impacts of guideline implementation in individual and population outcomes (Damschroder et al. 2009; Kirk et al. 2016).

Undoubtedly these efforts to amend existing problems in guideline development have had a very positive impact, but questions arise as to whether amendments to the EBM model will suffice to meet the current challenges or a conceptual change is required.

Should the EBM model be changed for improving guideline development?

In the past few decades there has been a substantial questioning of the applicability and practicality of the EBM framework and a ‘one size fits all’ approach to clinical practice (Charles et al. 2011; Rosner, 2012; Salvador-Carulla et al. 2013; Greenhalgh et al. 2014, 2015; Fernandez et al. 2015; Bingeman, 2016). A number of new guideline development methods that follow different approaches have emerged in the recent years. For example, Yelovich has advocated an alternative perspective of the patient–physician relationship as a meeting of the professional expert and ‘experiencer’, with authentic recognition and value of the patient's tacit knowledge (Yelovich, 2016). Lukersmith et al. have proposed an alternative method to develop recommendations using framing with an international classification of functioning disability and health to ‘pack’ different sources of scientific and context knowledge (Lukersmith et al. 2016). Ivers and Grimshaw have suggested the setup of ‘implementation laboratories’ to utilise and apply learning, context and delivery knowledge as evidence to inform health system policy (Ivers & Grimshaw, 2016). Ostuzzi and colleagues have used a ‘bottom-up’ approach of expert knowledge to develop recommendations on the complexity of prescribing psychotropic medications (Ostuzzi et al. 2013). There have been calls for guidelines to adopt the biopsychosocial framework of health, include patient reported outcome measures (van Dulmen et al. 2015); and the incorporation of a context knowledge base on use and costs of people with intellectual disability for evidence informed policies (Salvador-Carulla & Symonds, 2016).

The need for a philosophy of science approach and an extended model of scientific knowledge to guideline development

From our perspective the traditional model of EBM is necessary but not sufficient for developing practical recommendations for guiding decision making under conditions of complexity. It requires a critical revision of its foundation that draws on three major challenges for the success of guidelines as implementation tools:

The difficulties of adapting the EBM framework to a systems thinking and complexity approach

Systems thinking and complexity approaches are essential in health systems research, guidelines for complex interventions (e.g., community-based care), complex target health conditions (e.g., mental and developmental disorders, multi-morbidity) and complex practice contexts where decisions are made under conditions of high uncertainty such as in primary care and mental health care (Sturmberg et al. 2010; Salvador-Carulla et al. 2013; Huckel Schneider et al. 2016 (in press)). Complexity in clinical situations requires reasoning and judgment that may rely on broader sources of knowledge than the unidimensional guideline recommendation. In addition, a systems thinking approach is not intended to generate a single and definite solution to a PICO question, but to add new information to the knowledge base to improve decision making from a formative/summative perspective to organisational learning (Hargreaves, 2010; Snilstveit et al. 2012).

The lack of a philosophy of science approach to guideline development

We have revised this problem in a series of prior papers where we advocate for the extended multi-domain approach to scientific knowledge (Salvador-Carulla et al. 2014; Fernandez et al. 2015; Lukersmith et al. 2016). The definition of a series of key terms related to the development of this model is provided in Table 1. First, it is necessary to consider the differences between the three stages of scientific knowledge: discovery, corroboration and implementation (Schurz, 2014) and their contribution to guideline development. The traditional approach to EBM implicitly assumed that the analysis and ordering of the existing knowledge base of discovery and corroboration could be automatically translated, implemented and applied in practice. However, knowledge translation and implementation research is emerging in response to the need to identify the determinants, processes and outcomes of translating (corroboration) research into practice (Rabin et al. 2008; Peters et al. 2013).

Table 1.

Concepts and definitions of an extended multi-domain approach to scientific knowledge^a

Basic concepts	Definitiona
Scientific knowledge	A fluid mix of framed evidence and expertise acquired by means of standardised methods of research following the principles of commensurability (definitions of units of analysis that can be compared like-with-like), transparency for corroboration (including replicability and falsifiability) and transferability (including generalisability to broader contexts)
Evidence	The part of scientific knowledge based on contextualised information from facts and data, and which is analysed using quantitative approaches alone or combined with qualitative methods to generate inferences using mainly deductive reasoning, but also non-deductive logical reasoning (induction and abduction)
Expertise	The part of scientific knowledge based on expert know-how, understanding, experience and insight and on the perceived experience of a phenomenon. Experiential knowledge is here considered part of this category together with professional (expert) knowledge
Stages of scientific knowledge
Discovery	Generation of new and relevant scientific evidence mainly using experimental approaches and deductive inference. Discovery can also be generated by consilience using inductive inference
Corroboration	Justification of the new scientific evidence by determining the degree of confirmation given the acceptability of experimental and observational data using logical induction. It requires transparency of prior information and uses quantitative techniques for ordering available evidence (e.g., meta-analysis), and qualitative techniques for reaching expert consensus (inter-subjectivity)
Implementation	Factual application of corroborated scientific knowledge to real life practice and policy and to controversial cases. Under conditions of uncertainty and non-monotonicity it requires expert knowledge and incorporates abduction and means-end inferences logical reasoning in the decision-making process
Types of logical reasoning
Inference	Process of deriving logical conclusions from premises known or assumed to be true.
Deductive	Inferences from general instances to a specific conclusion. Is the main necessary inference, i.e. the certainty of the explanation can be derived from the certainty of the premises
Inductive	Inferences from specific instances to general conclusion or explanation. General statements are made based on specific observations
Abductive	Inference to the best explanation. It needs a prior knowledge-base to select the best or the most plausible explanation
Means-end inferences	Relates fundamental norms to the means to achieve a pre-determined end. This requires experts to decide which is the best or optimal mean from a set of alternatives to achieve the final goal
Domains of scientific knowledge
Experimental evidence	Information acquired conducting an experiment to test a formal hypothesis using deductive reasoning (e.g., Randomised control trials)
Observational evidence	Information acquired by the standard recording of phenomena under natural conditions and analysed mainly using inductive reasoning (e.g., cohort studies, surveys). Ecological evidence refers to observations of the phenomenon gathered in a clearly defined area and environment
Contextual evidence and knowledge	Context refers to the totality of circumstances that comprise the milieu of a given phenomenon. In health care it includes all sources of evidence of the local system: geography, social and demographic factors, other environmental factors, service availability, capacity, use and costs. It also includes legislation and expertise on the milieu (e.g., the historical account of the current state of the art)
Expert knowledge	A set of formalised know-how, understanding, experience and insight in a defined area of knowledge which is informed, contextualised, stable, consistent and connected. It is elicited using qualitative approaches alone or combined with quantitative methods to generate means-end inferences and non-inferential knowledge to complement evidence
Experiential knowledge	The knowledge and understanding of the health condition, intervention and context derived from exposure as a patient, consumer or carer
New types of scientific research relevant for guideline development
Framing of scientific knowledge (FSK)	A group of studies of ‘expert knowledge’ specifically aimed at generating formal scientific frames. FSK studies contribute to the formulation of research questions, to understand and to represent complex phenomena and to guide decision making under conditions of uncertainty and insufficient evidence. FSK studies generate formal scientific frames that could be used to analyse and to interpret complexity in health sciences, particularly in public and health systems research. FSK includes scientific declarations and charts, conceptual maps, classifications and recommendations of clinical guidelines

^aThese formal definitions are partly based on the unified approach to the philosophy of science (Schurz, 2014) and have been described in Salvador-Carulla et al. 2014, Salvador-Carulla & Symonds 2016 and Lukersmith et al. 2016.

Second, it is important to provide operational definitions of the different domains of scientific knowledge (experimental evidence, observational evidence, contextual evidence, expert knowledge and experiential knowledge), the areas of scientific research relevant to understand recommendations (e.g., framing of scientific knowledge as a new research area of expert knowledge) (Salvador-Carulla et al. 2014), and logical inferences apart from deduction and induction, and which are extremely relevant to recommendations’ development such as abduction and mean-ends inferences (Table 1).

Alternatives to the EBM pyramid for ranking the hierarchy of research evidence

The traditional EBM hierarchy (Howick et al. 2011) graphically portrayed as the EBM pyramid places a higher value on deductive inference from experimental evidence although systematic reviews rely heavily on induction. In the EBM model, randomised controlled trials are considered the highest quality, whilst observational evidence is de-valued, and context, expert ‘opinion’ and consumers’ experience are factually excluded as relevant sources of knowledge in the systematic reviews and the subsequent recommendations (Rosner, 2012). Expertise is not only important as a source of knowledge but also to generate recommendations. The ‘gold standard’ for guideline methods requires the inclusion of expert clinicians and experiencers consumer groups in the guideline development team. Yet, their contribution to the recommendations is typically not made transparent. Paradoxically the recent attempts to refine and improve the pyramid model (e.g., EBHC pyramid 5.0) (Alper & Haynes, 2016) insist on the analysis of the quality of evidence based on its internal validity and not on its external validity which is critical in complex questions which are context-dependent (Fernandez et al. 2015).

It is clear that the methods to formulate guideline recommendations need to accommodate the complexity, uncertainty and variability of health care practice. In our opinion, there needs to be a new framework for the development of guidelines if they are to be ‘fit for purpose’, feasible, useful and assimilated into practice. The framework should incorporate all the advances in the application of the knowledge-base of discovery to clinical practice made by EBM, but it also requires repeatable and explicit methods to integrate other scientific knowledge types described in Table 1. As guidelines are implementation tools, a new framework must consider scientific knowledge derived from implementation research (context and impact analysis). We have taken the first step and developed a preliminary model to frame an extended multi-domain approach to scientific knowledge that is graphically represented as a ‘Greek temple’. Figure 1 illustrates the movement from the unidimensional approach to broader extended multi-domain model of scientific knowledge to generate clinical guidelines and recommendations.

Fig. 1.

Moving from the pyramid model of scientific knowledge to the General framework: Greek temple model of scientific knowledge.

The five columns of scientific knowledge used across discovery, corroboration and implementation research, represent the domains of scientific knowledge described in Table 1. We consider that although experimental evidence derived from discovery and corroboration is a basic pillar for developing guidelines, other domains should be considered in guideline development and they cannot be merged together in a unidimensional ranking. Separate analyses should be provided to summarise the experimental knowledge and the observational knowledge using the criteria already available (e.g., EBHC pyramid 5.0) (Alper & Haynes, 2016), to provide a scientific framing together with information on context analysis, expert knowledge on appropriateness/adequacy, and the experiential knowledge on acceptability. For example, electroconvulsive therapy may show a high-efficacy and cost-effectiveness for major depressive disorder (Milev et al. 2016), and at the same time high differences in local availability (context), and in the judgment by expert professionals (appropriateness) and by consumers (acceptability). It is extremely difficult to display this complexity in a single dimension. This case may be better represented by a multi-dimensional profile than a single statement followed by the strength of the recommendation. However, much more work needs to be done to develop and test transparent, rigorous and repeatable guideline development methods to integrate and package all scientific knowledge on the topic of concern in an acceptable, applicable and practical way.

Case example: Canadian consensus guidelines for the primary care of people with intellectual and developmental disabilities (IDD)

The ‘Canadian Consensus Guidelines for the primary care of people with developmental disabilities’ were developed in 2006 (Sullivan et al. 2006), revised in 2011 (Sullivan et al. 2011) and are currently being revised in 2016. We describe the complexities of the health condition and practice context; a problem intervention; and an outline of the guideline methods. The case example is from the 3rd version of the Canadian Consensus Guidelines on the primary care of adults with developmental disabilities, 2016 updates. The IDD Canadian Guidelines for primary care providers propose preventative health care recommendations that they can and should implement during regular health care assessments of adults with IDD. However, these guidelines illustrate the challenges associated with identifying scientifically robust recommendations using an EBM approach for a population with often complex health conditions, such as adults with IDD.

Health condition and practice context:

‘Intellectual developmental disorder’ or ‘IDD’ shows unsolved differences in its terminology, standard diagnostic criteria and assessment (Bertelli et al. 2016a, b); as well as in its care delivery system, planning and policy (Salvador-Carulla & Symonds, 2016). Adults with IDD often have health conditions that differ from those of the rest of the population, but there are few PICO-type studies for this group of patients. Most of the scientific knowledge regarding this population is rated as low-quality expert opinion within the EBM framework. Even when the same health conditions are present in the IDD population as the general population, evidence-based recommendations are not necessarily inclusive of adults with IDD because of the wide variability from person to person of factors such as level of disability (i.e., mild to profound) and aetiology, that could affect the outcome of interventions. Moreover, many adults with IDD experience co-morbid physical, behavioural and mental health conditions (for which separate evidence-based guidelines might exist but do not include the IDD population). The practical usability of guidelines for adults with IDD is highly limited due to conflicting recommendations and the risk of engaging in multiple recommended interventions (e.g. the use of medications in comorbid and complex cases).

A clear example is provided by the use of antipsychotic medications for problem behaviours in persons with IDD. Antipsychotics are often prescribed by family physicians in Ontario (Cobigo et al. 2013). Available studies provide insufficient evidence to support such off-label prescribing practices (Deb et al. 2015). In addition, EBM neither elucidates factors that influence clinical decisions to start or continue such prescriptions nor does it assist clinicians or consumers in determining, whether the benefits of prescribing antipsychotic medications outweigh its burdens and risks (e.g., sedation and metabolic syndrome).

Problems in the implementation of the IDD Canadian guidelines

Authors updating the IDD Canadian guidelines recognised several methodological problems in the former version (2011). First, most of the recommendations were based on expert consensus only, which is regarded as the lowest quality of evidence in EBM. Second, the strength of a recommendation was assumed to be the same as the quality of the best supporting research evidence. For instance, this assumption was not conducive to judging that level I evidence (of the highest quality in EBM) could ever warrant a weak recommendation for action, because of factors relating to the context of care or patient or caregiver preferences and values, for example, just as level III evidence (of the lowest quality) might sometimes warrant a strong recommendation. As a result, there was very little in the 2011 Guidelines that could be proposed as a strongly recommended action. Third, for the level III evidence on which most of the 2011 Guidelines were based, the authors did not consider whether or not the experts had based their views on experiences from comparable medical contexts or had taken into account the complexities of caring for a heterogeneous patient population among adults with IDD with varying health needs, experiences, preferences and values. Fourth, the link between recommendations and framing values, such as a commitment to patient-centred care, adaptations to practices to accommodate the specific needs of people with IDD, and a commitment to supported decision-making, was not required by an EBM approach to developing the guidelines.

The implementation rate of any regular preventive care assessment of adults with IDD in primary care settings in Ontario, Canada, was found to be low (Ouellette-Kuntz et al. 2013) . In a subsequent study regarding implementing health checks for people with IDD in different primary care settings, many system and other factors that influence their successful implementation were identified (Durbin et al. 2016). The EBM-knowledge on which prior versions of the IDD guideline recommendations were based was silent on these contextual factors, and the authors sought to explicitly consider other types of knowledge and clinical context in the updated version of the guidelines.

Outline of the methods for the development of the 3rd version of the IDD Canadian Guidelines

Given these and other limitations of a strictly EBM approach to guideline development, several alternatives were undertaken. First, the authors supplemented the knowledge base for guidelines by incorporating other sources of knowledge related to context, expert knowledge, and consumer and carer knowledge, as described above. These sources of knowledge were especially useful for deciding which guidelines to revise, delete or add based on the cumulative experience of Canadian primary care providers who had implemented the 2011 Guidelines. By making these additional categories of knowledge explicit, the authors allowed for some recommendations for action for which there was no ‘evidence’ according to the EBM model. It is important to note that the addition of context knowledge was made possible by the previous work of the Healthcare Access Research and Developmental Disabilities (H-CARDD) programme. This is a partnership of scientists, policymakers, and clinicians which uses administrative data to provide information on the health of a cohort of adults with IDD relative to other adults in Ontario. The key findings of H-CARDD have been published in the Atlas of the Primary care of Developmental Disabilities in Ontario (Lunsky et al. 2013; Salvador-Carulla & Symonds, 2016).

Second, the authors ranked quality according to different criteria for each of the types of knowledge profiled. For example, empirical studies were ranked ‘level I’ if they were randomised control trials or systematic reviews and ‘level II’ for other types of studies, whereas expert knowledge was classified as ‘level I’ if they had been generated by an explicit consensus process and ‘level II’ if they had not. By profiling the different domains of knowledge considered (rather than simply ranking the quality of knowledge as an aggregate), the authors enabled clinicians to make their own judgments regarding recommended actions based on what they deemed most relevant to particular patients and their context of care.

Third, the authors took a complexity approach to their proposed action for each guideline by providing ‘advice’ or guidance that is relevant for individual patient care rather than being based abstractly on generalised ‘evidence’. Three different levels of recommendation have been established for every domain:

• (1)‘Recommend’: always apply. The primary care provider should regard the guideline as a basic standard of care in Canada.
• (2)‘Consider’: indicated to primary care providers that it should be applied at the physician's discretion. The professional would need to take into account factors specific to each patient and context before deciding to implement the guideline.
• (3)‘Aspire’: authors endorsed the action as an ideal/future standard of care that should be applied if possible but acknowledged that advocacy and changes to contexts and systems of care might be necessary for clinicians to have the capacity and resources to apply such a guideline routinely.

By distinguishing among these types of advice, the authors were able to reduce the number of guideline recommendations as basic standards of care. They were also able to add new guidelines that could be regarded as aspirational but were excluded from the 2011 Guidelines (which focused only on what was judged to be attainable at that time within the Canadian health care system). The authors were also able to flag those guidelines for which additional particular factors in varying contexts of care and in different patients would play a significant role in clinical decision making.

Finally, framing values were incorporated into the guidelines themselves by adding to the General Issues section of the 2011 Guidelines some guidance regarding approaches to care. In addition, the authors revised each guideline with the view of ensuring consistency with these framing values. They also sought to promote good concrete clinical decisions by acknowledging the importance of considering additional particular factors for each patient prior to implementing the action they were to consider, such as the patient's strengths and vulnerabilities in decision making, and regard for patient and caregiver input and participation in decisions concerning the proportion of benefit to harm for the patient of the intervention being considered.

Conclusions and practical implications

Clinical guidelines are implementation tools aimed at supporting decision making in the real world and in different contexts to enhance health care quality, maximise resources and innovation. Adopting a systems thinking approach may lead to the development of new methods and a different arrangement and integration of scientific knowledge that may reframe and broaden the approach provided to guideline development in the past two decades.

Guidelines should incorporate information from all five domains of scientific knowledge (observational, experimental, contextual, expert and experiential knowledge) in their systematic review section. The section of recommendations is actually a separate type of knowledge based on expertise (framing of scientific knowledge) that should be generated following a transparent qualitative method. Recommendations could be provided as a profile of domains following a ranking that is not only based on the strength of the recommendation following the example of the IDD Canadian guidelines. Guidelines updated over time should be considered in a higher-order category of clinical guidelines and they should incorporate information on their implementation impact during the previous phase.

Financial Support

This research received no funding.

Conflict of Interest

The authors declare no conflict of interest.

Author Contributions

Luis Salvador-Carulla and Sue Lukersmith contributed equally to the concepts and drafts of the manuscript. William Sullivan contributed the case example and manuscript revisions.