Evaluating Complex Healthcare Systems: A Critique of Four Approaches

Abstract

The purpose of this paper is to bring clarity to the emerging conceptual and methodological literature that focuses on understanding and evaluating complex or ‘whole’ systems of healthcare. An international working group reviewed literature from interdisciplinary or interprofessional groups describing approaches to the evaluation of complex systems of healthcare. The following four key approaches were identified: a framework from the MRC (UK), whole systems research, whole medical systems research described by NCCAM (USA) and a model from NAFKAM (Norway). Main areas of congruence include acknowledgment of the inherent complexity of many healthcare interventions and the need to find new ways to evaluate these; the need to describe and understand the components of complex interventions in context (as they are actually practiced); the necessity of using mixed methods including randomized clinical trials (RCTs) (explanatory and pragmatic) and qualitative approaches; the perceived benefits of a multidisciplinary team approach to research; and the understanding that methodological developments in this field can be applied to both complementary and alternative medicine (CAM) as well as conventional therapies. In contrast, the approaches differ in the following ways: terminology used, the extent to which the approach attempts to be applicable to both CAM and conventional medical interventions; the prioritization of research questions (in order of what should be done first) especially with respect to how the ‘definitive’ RCT fits into the process of assessing complex healthcare systems; and the need for a staged approach. There appears to be a growing international understanding of the need for a new perspective on assessing complex healthcare systems.

Background

Complex treatment systems such as palliative care, public health, integrative medicine, rehabilitative medicine or traditional Chinese medicine, and interventions within those systems, are an important part of healthcare around the world. However, these approaches to healthcare are not always well served by the biomedical model of diagnosing, treating, understanding and evaluating diseases which emphasizes the evaluation of single-component interventions. The applicability of this model for investigating healthcare as it is actually practiced is limited. Hence, a broader perspective is necessary.

Complementary and alternative medicine (CAM) researchers have a particular interest in driving the debate about how best to assess complex healthcare systems, as they struggle with demands from regulators, insurers, purchasers, providers and patients for ‘evidence’ of effectiveness and efficacy in order to meet the standards of ‘evidence-based medicine’. The debate regarding these research design issues within conventional medicine has risen in parallel with the growing emphasis on team-based medicine and integrative medical teams, and, related to this, the increasing complexity of treatment interventions. In addition, there is increased recognition that explanatory (placebo-controlled) randomized clinical trials (RCTs) alone cannot adequately assess these interventions and their outcomes. Explanatory RCTs are conducted under conditions that are as controlled as possible and include the following characteristics: administration of a placebo to the control group in an attempt to hold all possible causal elements constant except for the intervention under investigation; standardization of inclusion and exclusion criteria; standardization of the intervention under investigation; randomized allocation of the participants to the intervention or the control group(s); blinding (allocation concealment) of the participants and investigators (if possible) (1).

A number of descriptive articles have attempted to explain how one might begin to assess these complex interventions. Each approach has developed its own language and concepts to describe the phenomena, making communication and consensus building difficult across approaches. For example, a variety of terminologies has been proposed to describe what appears to be essentially the same phenomenon. In this paper we use the term ‘complex healthcare systems’. We define these as complex interventions to improve or enhance health and well-being as well as to prevent disease. ‘Complex’ denotes the entangled interrelationships among multiple ‘active’ components of the intervention. In addition, it highlights that, in general, the effects of the ‘whole’ intervention or system are interactive rather than additive (2–4), with the potential that the whole is more than the sum of the parts. This reasoning reflects the theory that complex systems have an inherent self-organizing property and that the elements of complex systems themselves interact in such a way that through the interplay of the elements new properties emerge that cannot be seen when investigating only the component parts. In our view, both the human body and systems of healthcare have to be seen as complex, self-organizing systems that create new, emerging properties through the interplay of their component elements (5–9).

The purpose of this paper is to compare, contrast and critique four different approaches to the assessment of complex healthcare interventions or systems, also identified as ‘whole’ systems. Terms such as whole systems, complex systems, CAM systems and whole medical systems appear to describe similar concepts. However, this divergence in terminology reflects some unique features with respect to how each is defined and also the cultural context in which each arose. The present paper aims to bring some clarity to the field and helps to establish a broader awareness and understanding of the issues, as well as to facilitate interdisciplinary research.

Methods

An international working group of researchers met in 2005 in Tromsø (Norway) to further develop whole systems research methodology and identified the need to bring clarity to the emerging conceptual and methodological literature. In order to address this, a subgroup of seven people reviewed literature relevant to designing appropriate research methods to assess complex healthcare systems. They used the following criteria:

• Published documents describing approaches to the evaluation of complex systems of healthcare.

• Representing consensus statements from specific bodies or organizations.

• Written from an interdisciplinary/interprofessional perspective.

The group did not attempt to be comprehensive in this selection, but rather strove to identify a range of documents from a variety of countries to explore the diversity of approaches to the issue. Four approaches were identified at the meeting:

1. Complex interventions research (MRC, UK 2000) (10,11).

2. Whole systems research or WSR (International group, 2003) (2,3,12).

3. CAM systems research (NAFKAM, Norway 2004) (13,14).

4. Whole medical systems research (NCCAM, US 2005) (15,16).

After the original meeting in Norway, a fifth approach from the USA Institute of Medicine's Committee on the Use of Complementary and Alternative Medicine by the American Public was reviewed (17). While this fifth approach does identify the need for appropriate study design for ‘complex treatment packages’, they indicate that a detailed discussion is beyond the scope of their report. Thus, this fifth approach is not reviewed for this paper.

A minimum of two members of the subgroup independently completed a focused qualitative content documentary analysis of each approach (18,19). Key documentation associated with each approach was reviewed to identify underlying principles and assumptions, models for assessing complex healthcare systems, and specific strengths or weaknesses of the approaches. Individuals assessing each approach then met face-to-face until they reached consensus on the key themes from the documents and prepared to present the approach to the rest of the authors. An additional face-to-face discussion of each individual approach was held to identify commonalities and differences amongst the approaches reviewed. Preliminary findings were then submitted by the subgroup to the entire group at the international meeting for discussion and feedback. (The subgroup working on this project was one of three groups working on different projects as part of the international meeting.) The findings from this process of interpretation and synthesis, with minor modifications resulting from subsequent electronic communications, are provided below.

Results

The underlying principles, assumptions and approaches to research of each of the four approaches are described below and summarized in Table 1. The approaches are discussed in chronological order. Key similarities and differences are then highlighted.

Table 1.

Key research issues of the four approaches

Complex intervention research (MRC) (10,11)	Whole system research (international group) (2,3,12)	CAM systems research (NAFKAM) (13,14)	Whole medical systems research (NCCAM) (15,16)
Establish the theoretical basis of the intervention	Identify appropriate designs and analysis strategies	Describe the theoretical framework, and real life experiences within CAM	Acquire understanding of CAM systems and how they operate within their settings
Identify and describe the components of the complex intervention	Study the interactions of patients and practitioners	Monitor the safety of the CAM system	Document the benefits of some CAM treatments for selected health conditions
Describe a feasible protocol for comparing the intervention to an appropriate alternative	Study designs that reflect and/or assess the healthcare environment	Investigate the system effect of the whole CAM intervention	Elucidate mechanisms underlying successful multimodal treatments used in CAM
An intermediate goal is a ‘definitive’ randomized, controlled study, blinded where feasible	Understand the theoretical underpinnings of the intervention	Explore the specific effect of isolated components in a CAM intervention
Assess long-term and real life effectiveness and potential adverse effects	A research framework must be non-hierarchial, networked, cyclical, flexible and adaptive and hold qualitative and quantitative research methods in equal esteem	Assess the underlying mechanisms behind the effect of CAM

MRC, Medical Research Council, UK; NAFKAM, Nasjonalt Forskningssenter innen Komplementaer og Alternativ Medisin, Norway; NCCAM, National Center for Complementary and Alternative Medicine, USA.

Description of the Four Approaches

Complex Interventions Research

The Medical Research Council (MRC) of the United Kingdom define complex interventions as comprised of “a number of separate elements which seem essential to the proper functioning of the intervention although the ‘active ingredient’ of the intervention that is effective is difficult to specify… . The greater the difficulty in defining precisely what, exactly, are the ‘active ingredients’ of the intervention and how they relate to each other, the greater the likelihood that you are dealing with a complex intervention” (p2) (10).

An example of a complex intervention provided for this approach is a specialist multidisciplinary stroke unit. The Medical Research Council document states that a complex intervention is comprised of components (including behaviors; parameters of behaviors such as frequency and timing; and methods of organizing and delivering those behaviors) that may act independently and interdependently. The explicit goal of this approach is to reduce, if possible, the intervention to its ‘essential’ components so it can be assessed using a ‘definitive’ randomized controlled trial. In preparing for a definitive RCT, the report argues that it is essential to conduct pre-trial studies to assess components prior their use, to select the study design, define the intervention and assess potential outcome measures. Many different methods, such as qualitative research and observational trials, are proposed for these pilot studies (10).

The Medical Research Council perspective explicitly identifies a specific sequential and step-wise model described as a ‘continuum of increasing evidence’, which follows the conventional model of pharmacological testing and includes five phases as described in Table 2. The first pre-clinical phase, and also the second, modeling phase, may be skipped if the intervention is already in wide-spread use or if answers for public policy questions are needed urgently. Phase three, the use of exploratory trials focuses on testing one's ability to fully control all aspects of the intervention including varying different components to explore the effects this has on the intervention as a whole. These data are used to plan the randomized, controlled, blinded (if possible), definitive trial described as phase four of this approach. The final phase five focuses on long-term implementation, assessing real-life effectiveness and surveillance to identify rare or long-term adverse effects (10). Progression from one phase to another does not have to be linear, and there is scope for an iterative approach (11).

Table 2.

Comparison of two approaches with a stepwise approach to clinical evaluation

Phase	Complex intervention research (MRC) (10,11)	Phase	CAM systems research (NAFKAM) (13,14)
0	Theoretical (pre-clinical) phase; description (map) of elements from the literature	1	Describe clinical practice in widespread use
I	Modeling: understanding intervention and its possible effects	2	Assess safety in routine practice; data collection (quality, quantity) observational studies
II	Exploratory (pilot) trial(s): to assess feasibility and optimize design of main RCT	3	Assess the system effect (effectiveness of routine practice possibly using a pragmatic RCT)
III	‘Definitive’ RCT (explanatory or pragmatic)	4	Assess specific component effect (efficacy) principle method: explanatory RCT
IV	Long-term implementation: assess long-term and real life effectiveness and potential adverse effects ((likely to be an observational study)	5	Assess mechanisms of action

MRC, Medical Research Council, UK; NAFKAM, Nasjonalt Forskningssenter innen Komplementaer og Alternativ Medisin, Norway.

Whole Systems Research

Whole systems research (WSR) was first developed by an international group of researchers who met in Vancouver (Canada) in 2002. WSR is defined as encompassing “both the processes and the outcomes of complex healthcare interventions. WSR entails the intention to include conceptually, as part of the investigative context, all aspects of an internally consistent approach to treatment including its philosophical basis, patients, practitioners, setting of practice and methods/materials used. WSR acknowledges unique patient, family, community, and environmental characteristics and perspectives. The aim is to use appropriate research designs so that the system can be assessed within its explanatory model” (p. 33) (2).

This approach clearly identifies the importance of philosophical coherence and internal consistency when defining a whole system of healing. Examples of whole systems provided include traditional Oriental (or East Asian) medicine, naturopathic medicine, homeopathy, integrative medicine and biomedicine. A hallmark of this perspective is that it is explicitly non-reductionist and based on an underlying assumption that the effect of the whole system is greater than the sum of its parts. ‘Individual components of most whole systems are inseparable, complementary and synergistic and therefore WSR must not focus only on the elements that may initially be perceived as ‘active’ ingredients of a system’ (12).

In WSR, there is an explicit intention to include all aspects of an internally consistent approach to treatment as part of the investigative context (e.g. its philosophical basis, patients, practitioners, context of care, patient–provider relationship and methods/materials used.) This perspective suggests that randomized controlled trials may be limited in their ability to adequately assess whole systems and thus recommends a mixed methods approach that includes a range of relevant and holistic outcome measures. Proponents of WSR argue that programs of research, rather than single studies, are needed to study whole systems. This perspective does not identify a specific model for doing clinical research, but does provide fundamental components of what is called an ‘emergent Whole System Research framework’ (12).

CAM Systems Research

Nasjonalt Forskningssenter innen Komplementaer og Alternativ Medisin (NAFKAM, Norwegian National Research Center in Complementary and Alternative Medicine) in Norway describes an approach to the study of CAM systems that are already in widespread use (13,14). Based on the assumption that if a therapy is already widely used, then this approach argues that making sure it is safe becomes a first priority. Once concerns about safety are addressed, there is also a need to assess the ‘system effect’ (i.e. treatment response in patients due to whole CAM intervention as it is normally practised) as well as the ‘component effect’ (i.e. specific interventions in isolation). Although explanatory (placebo-controlled) randomized controlled trials are recommended for assessing component effects, multimethods including pragmatic randomized controlled trials are recognized as important for studying the system effect (14).

In contrast to WSR, the Norwegian National Research Center in Complementary and Alternative Medicine approach provides detailed steps for developing a program of research for assessing CAM systems that are already in widespread use. Although it is possible to enter the model at any stage, depending on how much information you already have (or what your research question is), this schema provides a framework for types of questions, and thus types of research, that should be conducted in order of priority (see Tables 1 and 2). Generally, this model recommends proceeding with research according to the stages of description, safety, system effect, component effect and underlying mechanisms. Because the therapeutic environment established by the CAM practitioner may have a clinical significant effect on healing processes and therapeutic responses, it is important in the Norwegian National Research Center in Complementary and Alternative Medicine model to elucidate and assess the patient's treatment response due to the total setting of the CAM therapy situation (system-effect) as well as the ‘specific interventions’ in isolation (component-effect) (13).

Whole Medical Systems

Whole medical systems are described by the National Center for Complementary and Alternative Medicine (NCCAM) in the United States as ‘complete systems of theory and practice that have evolved independently from or parallel to allopathic (conventional) medicine. Many are traditional systems of medicine that are practiced by individual cultures throughout the world’ (15). Examples of whole medical systems include: traditional Chinese medicine, homeopathy, naturopathy as well as systems developed by Native American, African, Middle Eastern, Tibetan and Central or South American cultures. Research of these systems is identified as problematic because of the underlying assumption that they are neither derived from nor have they adopted an evidence-based scientific perspective because interventions are usually individualized and multimodal (16). This approach explicitly identifies the need to assess the effects of the whole system in its entirety as it is commonly practised, which, it is argued, will require a multidisciplinary team.

The NCCAM approach perspective does not provide specific guidelines or steps for doing research, but does identify specific goals for research (see Table 1). Although these goals are not explicitly ordered by priority of importance, they may be suggestive of a program of research that moves from attempts to understand the system as it operates in its real-world setting, documenting potential health benefits and then elucidating mechanisms (16).

Congruence and Divergence

Common Themes

All four approaches acknowledge the inherent complexity of many healthcare interventions and recognize that the pharmacological RCT model of doing research is not sufficient to assess these. The need to describe and understand the components of complex interventions in context (as they are actually practiced) is consistently recommended as an early phase of the research process. Every approach explicitly identifies the necessity of using mixed methods, including RCTs (explanatory and pragmatic) and qualitative methods. Most suggest a multidisciplinary team is needed to achieve the range of methodological techniques required to conduct the kind of research programs envisioned. Most of the approaches also explicitly note that methodological developments in this field can be applied to both CAM and conventional therapies. The approaches describe ways to investigate complex systems in general, irrespective of the content of the systems.

Divergence

The one obvious difference among the four perspectives is the use of different terms to describe what appear to be largely overlapping concepts. ‘Whole systems’, ‘whole medical systems’, ‘CAM systems’ and ‘complex interventions’ clearly have much in common including the recognition of multiple components that interact and whose effects are not merely additive. At some points, the Medical Research Council statement appears to assume an additivity of the elements of complex interventions (which implies an underlying linear assumption), but in other places the statement that the parts ‘interact’ can be interpreted to mean that the ‘whole is greater than the sum of the parts’ (or that there is an underlying multiplicative assumption). So although different terminology is used, the underlying assumptions are quite similar across all four perspectives making this a relatively superficial difference.

Other differences of note are the indigenous cultural emphasis of the NCCAM approach of whole medical systems and the Norwegian National Research Center in Complementary and Alternative Medicine's focus on CAM interventions that are in widespread use. The WSR approach is explicitly identified as being applicable to both CAM and conventional medical interventions. In contrast, the Medical Research Council perspective focuses on conventional medical examples and although it implies it can be used to assess a range of different interventions, CAM examples are not explicitly mentioned.

Perhaps the most important difference across the approaches is the prioritization of research questions (in order of what should be done first) and whether a specifically staged approach is desirable. WSR specifically refutes the notion that a step-wise approach to clinical evaluation is necessary or desirable. This perspective argues for a cyclical and flexible research program (12). In the Norwegian National Research Center in Complementary and Alternative Medicine perspective (and the NCCAM perspective assuming the order of research goals implies the order in which the research should be undertaken), the effects of the whole system/intervention as it is normally practised should be studied first, followed by investigation of how the individual components contribute to the whole and finally mechanisms of action. In contrast, the Medical Research Council perspective focuses on necessary pilot work to allow the design of a ‘definitive’ RCT if appropriate, leading to long-term implementation and surveillance.

This highlights a related difference: how the ‘definitive’ RCT fits into the process of assessing complex healthcare systems. WSR, the NCCAM and the Norwegian National Research Center in Complementary and Alternative Medicine appear to describe RCTs as one tool in a very large and diverse methodological tool kit. In contrast, the Medical Research Council has as its cornerstone the design and carrying out of a ‘definitive’ RCT, though a range of other methods are encouraged in other phases.

All four perspectives provide some direction on how to proceed with the assessment of complex healing systems. However, only two (the Norwegian National Research Center in Complementary and Alternative Medicine and the Medical Research Council) provide specific models that suggest distinct steps to use when developing a research program (see Table 2). The others make only general statements about the way to move forward with research. This creates a challenge for commissioners of funded research looking for guidance to design targeted competitions or for providing advice to peer-review committees. Further discussion and formalization of how to actually ‘do’ this research appropriately is required.

Discussion

The most significant finding was that the International WSR group, the US NCCAM, the Norwegian National Research Center in Complementary and Alternative Medicine and the UK's Medical Research Council all acknowledge the need to investigate complex healthcare systems as integral systems. Other recent documents, such at the Institute of Medicine Report from the USA, have also identified this need (17). There is considerable agreement that the classical pharmacological RCT model alone is not sufficient. Yet there does not seem to be universal agreement on what should be done instead. This may be partly because the perspectives analyzed here evolved from different cultural and social contexts to meet different needs. For example, the Medical Research Council guidelines were specifically developed from a health services perspective in order primarily to help researchers design higher quality RCTs. In contrast, the Norwegian National Research Center in Complementary and Alternative Medicine perspective was developed to guide a CAM research center's priority setting and research goals for therapies that were already in widespread use.

This raises the question: Is there a need (and is it possible) to develop a comprehensive, common set of guidelines for assessing complex healthcare systems? Or is diversity needed to match different social contexts and different funding agencies? For example, the WSR perspective clearly advocates a non-linear, iterative approach (12). Prescriptive evidence hierarchies of research methods may not be helpful, but explicit guidelines of what needs to be incorporated or considered in complex healing systems research may help to increase the quality of future research.

Terminology

Terminology is another area where clarity is needed. Every approach uses different terminology for what we have labeled complex healthcare systems. Our choice of terms is based on the fact that all perspective either use ‘complex’ as part of their name or as a key descriptor for what they are discussing. Healthcare is used to focus on health or healing interventions that are relevant to both CAM and conventional medical interventions. It encompasses both coherent treatments (e.g. homeopathy) and delivery models such as public health or integrative medicine. Finally, we chose the term ‘system’ (as opposed to ‘intervention’) because it seems to imply the interrelated and interconnectedness of components in a way that intervention does not. The term ‘whole’ is used by two of the perspectives, yet what is meant by a ‘whole’ (as opposed to a ‘partial’) system appears unclear. Is acupuncture a whole medical system or not? Is it only a whole system when it is studied within a traditional Asian medical context? It has been argued that an intervention (or system) can be complex without being ‘whole’ and that what constitutes ‘wholeness’ may be in the ‘eye of the beholder’. Clearly, conceptual clarity and more consistent terminology would facilitate interdisciplinary and international collaboration.

Future Directions

Our analysis is limited to four perspectives and, with the exception of the Medical Research Council framework, has an emphasis on CAM research (an emphasis that paralleled the focus of the WSR meeting from which this paper was derived.) However, this perspective is readily adaptable to all kinds of complex interventions. An important next step will be to review individual discipline literatures (e.g. psychology, nursing, primary medical care, psychotherapy, etc.) to inform and expand the dialogue begun here. Ironically, it appears that the discussion starting to develop in the field of medical research is relatively well developed in evaluation research, but that the arguments, discussions and results have, so far, not been integrated by medical researchers (20). The development of guidelines for assessing complex healthcare systems may further the evolving discourse on what constitutes good research and good evidence. Chiappelli et al. (21) begin this discussion in a paper exploring how the principles of evidence-based medicine can be applied to complex CAM interventions. They stress the need to assess whether the measures and design are congruent with the study question and argue strongly for the ways to accurately assess a wide range of evidence in any evidence-based review. In addition, further debate about the concept of ‘complexity’ and its relationship to emergent properties is needed.

Conclusion

Despite the use of different terminology, analysis of these four approaches suggests a growing international understanding of the need for a new conceptual framework for assessing complex healthcare systems. Multiple methods and integrated programs of research undertaken by interdisciplinary teams appear to be necessary. This field will benefit from additional international and interdisciplinary dialogue.

Competing Interests: None to declare.