Using systems perspectives in evidence synthesis: A methodological mapping review

Abstract

Reviewing complex interventions is challenging because they include many elements that can interact dynamically in a nonlinear manner. A systems perspective offers a way of thinking to help understand complex issues, but its application in evidence synthesis is not established. The aim of this project was to understand how and why systems perspectives have been applied in evidence synthesis. A methodological mapping review was conducted to identify papers using a systems perspective in evidence synthesis. A search was conducted in seven bibliographic databases and three search engines. A total of 101 papers (representing 98 reviews) met the eligibility criteria. Two categories of reviews were identified: (1) reviews using a “systems lens” to frame the topic, generate hypotheses, select studies, and guide the analysis and interpretation of findings (n = 76) and (2) reviews using systems methods to develop a systems model (n = 22). Several methods (e.g., systems dynamic modeling, soft systems approach) were identified, and they were used to identify, rank and select elements, analyze interactions, develop models, and forecast needs. The main reasons for using a systems perspective were to address complexity, view the problem as a whole, and understand the interrelationships between the elements. Several challenges for capturing the true nature and complexity of a problem were raised when performing these methods. This review is a useful starting point when designing evidence synthesis of complex interventions. It identifies different opportunities for applying a systems perspective in evidence synthesis, and highlights both commonplace and less familiar methods.

Highlights.

What Is Already Known

• Systems perspectives provide a holistic way of thinking and allow the study of complex systems, that is, systems with many interdependent elements interacting dynamically and in a nonlinear manner.

• Researchers have been advocating applying systems perspectives in evidence synthesis to better understand intervention complexity.

What Is New

• Two categories of papers using a systems perspective were identified: (1) reviews using a “systems lens” to frame the topic, generate hypotheses, select studies, guide the analysis and/or interpretation of findings, and (2) reviews using systems methods to identify, rank and select elements, analyze interactions, develop models, and forecast needs.

• There were various forms of implementation within each of the two categories.

• Some of the methods may be unfamiliar to evidence synthesists, including systems dynamics, agent‐based modeling, and soft systems methodology.

Potential Impact for RSM Readers Outside the Authors' Field

• This review is a useful starting point to help reviewers plan a review of complex interventions using a systems perspective. It identifies different opportunities for applying systems perspectives in evidence synthesis and highlights commonplace and less familiar methods for implementation.

1. INTRODUCTION

Reviewing complex interventions is challenging because they often include a large number of elements and complex pathways. ¹ The 2019 edition of the Cochrane Handbook for Systematic Reviews of Interventions suggests that three perspectives on intervention complexity are possibly applicable in systematic reviews: (a) a focus on the components of an intervention; (b) a focus on interactions between components of an intervention and/or between an intervention and other aspects (e.g., participants, context); and (c) a focus on the system within which the intervention is introduced (i.e., impact of an intervention on the dynamics of a system or of the system on an intervention). ² Several methods to address the first two perspectives have been suggested in the literature and are currently used, such as network meta‐analysis ³ and causal chain analysis. ⁴ However, although researchers have been advocating using a systems perspective to review interventions complexity for some time, ² , ⁵ , ⁶ , ⁷ , ⁸ only a few papers on systems methods in evidence synthesis have been published.

A system is defined as “an interconnected set of elements that is coherently organized in a way that achieves something.” ⁹ This definition highlights three important components of a system: (a) its elements (i.e., entities that characterize and make up a system), (b) its interconnections (i.e., the way elements feed back into, and are related to, each other) and (c) its purposes (i.e., what a system does). A system cannot be determined or explained by its elements alone since it is viewed as a whole that is more than the sum of its parts. ⁹ A complex system has been defined as a system that has a large number of interdependent elements and includes several properties such as nonlinear causation, emergent outcomes, feedback loops, and coevolution. ¹⁰ , ¹¹ , ¹² , ¹³ Complex systems can involve several actors from different organizations, sectors, and levels (e.g., micro, meso, macro). The complex adaptive system, a special case of complex system, consists of a system that can evolve and adjust over time. ¹⁴ Table 1 defines some characteristics that have been attributed to complex systems.

TABLE 1.

Attributes of complex systems

Attributes	Definition
Adaptation	The system adjusts and readjusts to changes in the environment over time. Similar concepts: dynamic, evolution, and constantly changing.
Coevolution	A system influences and is influenced by other systems. Similar concepts: dependencies and tightly linked.
Counter‐intuitive	Cause and effect are often distant in time and space. Effective solutions can work in some settings but not in others.
Emergence	Spontaneous, unplanned, or unpredictable behaviors arise from the interaction within or between elements of a system.
Feedback loops	Situations in which elements reinforce or balance other elements and influence subsequent actions.
System history/memory	The history of a system influences its nature and affects its evolution.
Multiple interacting elements	Complex systems are characterized by a large number of elements that are interconnected. These elements can be known or unknown.
Multiple agents	Different actors are involved within the system to contribute to solution making.
Non‐linearity	Not all relationships within a system can be arranged along an input–output proportional line.
Resistant to change	Failure to change may occur because the complexity of the system overwhelms one's ability to understand it and seemingly obvious solutions may fail or worsen a situation.
Self‐organization	The dynamics of a system arise spontaneously from its internal structure.

Source: Definitions adapted from: Petticrew et al., ⁶ Brainard and Hunter, ¹⁰ Rutter et al., ¹³ Plsek and Greenhalgh, ¹⁶ Sturmberg et al. ³⁰ Wilkinson et al. ³¹ De Savigny and Adam, ⁸⁰ Penney et al. ⁸² Richardson et al. ⁸³

Systems perspectives involve a holistic way of seeing the world. ¹⁵ Systems researchers are interested in understanding how the different elements of a system interact with each other and how these influence the system's behaviors. ⁹ , ¹⁶ In a systems perspective, interventions are seen as nested within a wider system that cannot be fully understood by only using linear causal models and by examining their components in isolation. ⁴ , ¹³ , ¹⁷ A systems perspective focuses on the context in which an intervention is introduced, the relationships and interactions between the actors and its context, and the changes within the systems itself. ⁶ Thus, when analyzing a complex system, it is necessary to analyze the dynamics of the system in order to forecast its possible evolution in the future. ¹⁸ For example, the COVID‐19 pandemic can be seen as a major perturbation in our complex biological and social systems with multiple and multifaceted elements that interact with one another in nonlinear and unpredictable ways. ¹⁰ , ¹³ The decisions made to control the pandemic do not follow a linear process, but are the result of feedback loops, such as policy responses to evidence on the spread of the virus. For example, the success of policies that recommend the wearing of face coverings, are not only the result of deterministic processes—such as the degree to which the covering prevents droplets traveling away from an individual. They are also influenced by shifting social and political norms that, in this case, mediate and moderate whether face coverings are worn at all. Moreover, there are complex social interactions between one policy—face coverings—and people's responses to other social policy, such as social distancing regulations. A systems perspective aims to permit the investigation of these types of multiple and interacting elements.

Systems perspectives have evolved since the 1950s. ¹⁹ A historical overview of different systems approaches has identified three main waves: hard, soft, and critical systems approaches ¹⁵ , ¹⁹ , ²⁰ (see Table 2 for a summary of their main characteristics). The first wave contains hard systems approaches that were mainly developed during the 1950s and 1960s in the fields of operational research and systems engineering. The aim of these approaches is to “understand how things are connected, by what, to what and with what consequence.” ²⁰ Hard systems approaches often use computer simulations to build models to represent the interaction between components of a system. During the 1970s, a second wave emerged with the development of soft systems approaches to explore the implication of different perspectives within a system on the interrelationships between elements. ¹⁵ They “put emphasis on dialogue, conflict resolution, mutual appreciation, and inter‐subjective construction of meaning.” ¹⁹ The purpose of doing this is to enable communication between different groups of individuals in order to identify potential solutions to improve the situation. ¹⁵ By the 1980s, a third wave is seen with the development of critical systems approaches to address the need to be critical about defining system boundaries and to establish boundaries within which critique can be conducted. ¹⁵ Three core values are essential in critical systems approaches: critique (e.g., being critical about the chosen methods and theories), emancipation (e.g., freeing the systems and its individuals from oppression and coercion), and pluralism (e.g., using a combination of methodologies). ²¹ Also, critical systems approaches are interested in addressing coercive contexts, especially contexts of marginalization, conflicts, ethical issues, and power relations. ²¹ , ²²

TABLE 2.

Main types of systems approaches

Approach	Ontology	Epistemology	Teleology (purpose)	Examples of methods
Hard	Realism	Postpositivism	Control Enables making sense of relationships between elements	Agent‐based modeling Network analysis Systems dynamics
Soft	Relativism	Constructivism/Interpretivism	Appreciation Enables communication and engaging contrasting perspective	Inquiring system design Strategic assumption surface testing Soft systems methodology
Critical	Relativism	Constructivism/Critical idealism	Emancipation Enables reconciliation of ethical issues and power relations	Community operational research System of systems methodologies Systemic intervention

Source: Adapted from Reynolds. ¹⁵

Systems perspectives have mainly been applied in primary research. For example, various systematic reviews of primary research that used systems perspectives can be found in the literature for a wide diversity of fields such as public health, ²³ education, ²⁴ and management. ²⁵ More recently, researchers have been advocating applying systems perspectives in evidence synthesis to better understand intervention complexity. ⁶ However, it is still not clear how systems perspectives might be applied in evidence synthesis and to what extent systems perspectives provide different analytical possibilities compared with existing methods addressing “complexity.” ¹ Thus, the aim of this project was to understand how and why systems perspectives have been applied in evidence synthesis.

2. METHODS

A methodological mapping review was performed. A methodological review synthesizes information on the conduct, analysis, and reporting of research methods. ²⁶ A “mapping” of the literature consists of collating, describing, and cataloging available evidence on a topic. ²⁷ A descriptive map provides information of the available studies across different topic areas. ²⁸ In this review, this allowed us to identify and describe the system approaches used in evidence synthesis.

2.1. Search strategy

Inspired by existing search strategies used in systematic reviews on systems perspectives of primary research, ¹⁰ , ²³ , ²⁹ , ³⁰ , ³¹ one author drafted a search strategy in consultation with an information specialist. It was then adapted for seven databases that collectively contain research in healthcare, social sciences, and business: Medline (OVID), Embase (OVID), PsycInfo (OVID), CINAHL (EBSCO), Social Sciences Citation Index (Web of Science), Public Health Database (ProQuest), and ABI/INFORM (ProQuest). Key terms on systems perspectives were combined with terms for evidence synthesis (see Table 3, e.g., the search strategy in Medline). The search was carried out from inception of each database to May 2019. This was supplemented by focused searching of three online search engines during November 2019: Google Scholar, Microsoft Academic, and BASE (Bielefeld Academic Search Engine). The aim of these later searches was to boost the identification of evidence synthesis outside health that could have been missed as the earlier databases predominantly covered health research. Search terms for the supplementary searches included: “systematic reviews” and “complexity theory”; “systematic reviews” and “systems thinking,” excluding health. All potentially relevant records were transferred to EPPI‐Reviewer ³² for eligibility assessment, data extraction, and synthesis. During the data extraction of included papers, a search in Google Scholar was performed to identify companion papers of included studies, that is, papers on the same study published elsewhere (such as Part 1 and Part 2 papers, or dissertations).

TABLE 3.

Search strategy (Medline)

Concepts	Search terms
Systems perspective	1. (“complexity science?” or “complexity theor*” or “system? approach*” or “System? change*” or “System? model*” or “System? perspective?” or “System? science?” or “System? thinking” or “System* theor*” or “Nonlinear* system?” or “Non linear* system?” or “Multi agent system?” or “Multiagent system?” or “agent based model*” or (Complex* adj2 system?) or (System? adj dynamic*) or (chaos adj1 theory)).ti,ab,kw.
	2. Nonlinear dynamics/
	3. Systems theory/
	4. Systems analysis/
	5. 1 or 2 or 3 or 4
Evidence synthesis	6. ((“review” and [integrat* or critical* or “mapping” or “comprehensive” or “evidence” or “research” or “literature”]) or ((“synthesis” or “systematic”) and (“evidence” or “research” or “review”))).ti.
	7. ((Systematic* adj2 review*) or “Meta analy*” or “Metaanaly*” or “metaethnograph*” or “meta‐epidemiological” or “meta‐regression” or “metasummary” or “data synthesis” or “evidence synthesis” or “metasynthesis” or “meta‐synthesis” or “narrative synthesis” or “qualitative synthesis” or “quantitative synthesis” or “realist synthesis” or “research synthesis” or “synthesis of evidence” or “thematic synthesis” or “bibliographic search” or “database search” or “electronic search” or “handsearch*” or “hand search*” or “keyword search” or “literature search” or “search term*” or “data extraction” or “narrative review” or “iterative review” or “realist review” or “scoping stud*” or “scoping review” or “systematic map*” or (mixed adj2 review*) or (mixed adj2 synthesis)).ti,ab,kw.
	8. “Systematic review”/
	9. Systematic reviews as topic/
	10. Meta‐analysis/
	11. Meta‐analysis as topic/
	12. 6 or 7 or 8 or 9 or 10 or 11
	13. 5 and 12

2.2. Eligibility assessment

Papers were included if they conducted an evidence synthesis in which the authors stated using a systems perspective in their review such as using a systems theory, systems method (e.g., as presented in Table 2), and/or developing a systems model. Several terms could be used to designate systems perspective such as systems sciences, systems thinking, systems theory, systems approach, complex adaptive system, complex sociotechnical system, and social‐ecological systems research. In this project, a broad definition of evidence synthesis was used: all papers that mentioned performing a literature review and described the methods used (e.g., search strategy, sources used, data extraction, and/or synthesis method) were considered for inclusion.

Papers were excluded if they were solely primary studies, nonempirical (e.g., opinion paper, editorial, commentary, summary), conference abstracts, protocols, and not written in English or in French. Moreover, conceptual or methodological papers and economic evaluations were excluded. Finally, reviews that aimed at understanding how systems perspectives were used in a specific field were excluded (e.g., review describing how systems science was applied in public health, ²³ review on health interventions that were informed by complexity science, ¹⁰ or review on systems dynamics modeling in entrepreneurship research ³³ ). These papers were considered reviews on systems perspective.

A two‐stage eligibility assessment process was followed starting with the screening of the titles and abstracts of records, and then examining the full‐text papers. One reviewer was involved in the whole assessment process. A second reviewer independently screened 5% of the titles/abstracts and discussion between the reviewers was held to clarify the eligibility criteria.

2.3. Data extraction and synthesis

The data extracted were: year, country of the lead author, topic, aim, number of studies included, reasons for using systems perspectives, methods used, software, systems visualization, data used, and limits/challenges encountered when using systems perspective as reported by the original authors. Two reviewers independently extracted data of two papers to pretest the data extraction form. Afterward, one reviewer performed data extraction for the remaining papers.

The literature was mapped and a thematic summary was produced because the aim of the methodological mapping review was to summarize what was reported in the included papers and not to reinterpret the published data. ³⁴ To produce the map, data on the reviews' characteristics (e.g., year, country, and topic), and systems approaches (e.g., systems methods, software, and systems visualization) were tabulated. For the thematic summary, data on the reasons for using systems perspectives and limits/challenges encountered when using systems methods were grouped into general categories by one reviewer and checked by a second reviewer.

3. RESULTS

A total of 3028 records were screened, 436 full‐text papers were assessed for eligibility, and 101 papers using systems perspectives were retained, representing 98 reviews (see Figure 1). The included reviews were published between 1986 and 2019, with most of them published after 2010 (n = 84). The first authors of the papers were affiliated with institutions from 20 different countries; United States (n = 38), Canada (n = 9), United Kingdom (n = 9), Germany (n = 7), Australia (n = 7), Brazil (n = 4), the Netherlands (n = 4), and Sweden (n = 3) were the most frequent countries. Most of the included papers addressed a topic in health (n = 49) followed by management and organizational studies (n = 15), child, adolescents, and family studies (n = 12), agriculture and environmental studies (n = 8), education (n = 6), information sciences (n = 4), physical activity (n = 2), urban and rural studies (n = 2). The number of included papers in the reviews ranged from 5 to 1012. Five were dissertations, of which two also published their literature review in a scholarly journal. Around a fifth of the included papers also used other data such as surveys, interviews, consultations with experts, and case studies (n = 20).

FIGURE 1.

Flow diagram

For this methodological mapping review, two general categories of papers using a systems perspective were identified. The first category comprises reviews using a systems lens and/or developing a systems model (n = 76) (see File S1). Several papers in this category mentioned using a systems theory or framework to guide the review. The cited theories and frameworks were: (socio‐) ecological systems theory developed by Bronfenbrenner (n = 28), complex adaptive systems (n = 9), complexity theory (n = 8), systems thinking (n = 6), systems theory or approach (n = 5), general systems theory by von Bertalanffy (n = 3), Rasmussen's Risk Management Framework (n = 3), family systems theory (n = 3), complex dynamic systems theory (n = 2), open system theory (n = 2), socio‐technical systems (n = 2), chaordic systems thinking (n = 1), health systems theory (n = 1), developmental system theory (n = 1), and Cynefin framework (n = 1).

3.1. System lens

A systems theory was used to frame a topic, define the concepts, identify variables, develop hypotheses, inform the selection of studies, code data, guide the analysis, and interpretation of results. The majority of these reviews used a systems framework to categorize or map the literature, to perform narrative synthesis, and to organize the findings. For example, Hong et al. ³⁵ used the ecological systems theory to examine how the individual characteristics, micro‐, meso−/exo‐, macro‐, and chrono‐systems level factors can influence or inhibit substance use among Asian American youth. Other synthesis methods were also mentioned such as content analysis (n = 4), ³⁶ , ³⁷ , ³⁸ , ³⁹ realist synthesis (n = 3), ⁴⁰ , ⁴¹ , ⁴² framework synthesis (n = 2), ⁴³ , ⁴⁴ meta‐synthesis (n = 2), ⁴⁵ , ⁴⁶ thematic synthesis (n = 2), ⁴⁷ , ⁴⁸ and meta‐analysis (n = 1). ⁴⁹ For example, McLain et al. ⁴¹ conducted a realist synthesis on fisheries' property regimes in which the socioecological framework was used as a theory for change framework to inform the context‐mechanism‐outcome (C‐M‐O) analysis. Also, one review used meta‐analysis to examine a systems model that represents the causal factors of occupational fatigue and their interrelationships. ⁴⁹ They performed meta‐analysis to explain the direction and magnitude of relationships among the factors.

In summary, the first category of studies applied a systems lens by using a systems theory or framework to organize or frame the review. They used traditional systematic review synthesis methods such as content analysis and statistical meta‐analysis, although they were guided by the systems theory in terms of data included, variables analyzed, and so on.

In contrast, the second category comprises papers that mentioned using a specific systems method to develop a systems model (e.g., systems dynamic modeling, soft systems approach) (n = 22). The descriptive analyses below focus on this latter category as they could provide different analytical possibilities compared with conventional evidence synthesis techniques.

3.2. Systems methods

Twenty‐two reviews used specific methods to explore the interrelationships of the elements of a system and develop a system model (see File S2). A diversity of methods to analyze the collected data was identified in the included papers; some used more than one method. They can be grouped into three main categories: (a) hard systems approaches; (b) soft systems approach and (c) other methods. The following will provide some examples of how these categories of methods were used in the included papers. The definitions of the methods are summarized in Table 4.

TABLE 4.

Definition of methods mentioned to study systems elements and their interrelations

Methods	Definition	Reviews
Agent‐based modeling	Modeling approach using “computer simulations to examine how elements of a system (agents) behave as a function of their interactions with each other and their environment.” 84	72
Analytic network process	Method for multi‐criteria decision making that allows ranking alternatives based on sets of elements, considering dependencies and feedbacks among factors, and producing forecasts. 70	61
“Cross‐over” modeling technique	Modeling technique that “forces the modeler to consider problems from the agent perspective, whilst providing insight into known and unknown variables such as the relationship between agents.” 69	69
Network analysis	Method that examines relationships and flows among elements of a system with visualization, description, and statistical modeling of networks. 84	73
Qualitative systems analysis	Involves two main steps: (a) identification and selection of system variables and (b) impact analysis, which consists of analyzing the mutual interactions among the variables. 85	67, 71
Soft systems methodology	Action‐oriented methodology to tackle problematic and messy situations. Four activities are suggested constituting a learning cycle process: (a) find out about a problematic situation; (b) make some relevant purposeful activity models; (c) debate about the situation and find changes that are desirable and culturally feasible; (d) take action to improve the problematic situation. To build models, the CATWOE analysis (Customers, Actors, Transformation process, Worldview, Owners, Environmental constraints) is suggested. 86	74, 75
Strategic flexibility analysis	A scenario analysis tool is used to assess the capability of a system “to respond to a dynamic environment through continuous changes and systemic actions” and to forecast future needs. 71	67, 71
Systems dynamic	Modeling approach using computer simulation to build models that represent the dynamic complexity of a system by understanding the “interplay of feedback loops, stocks, and flows that all occur within the bounded endogenous system.” 84	50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68
System of systems (SoS) framework	SoS is a complex system formed of several independent systems. Meta‐modeling is suggested to model the inter‐ and intra‐dependencies among and between each subsystem within a system of systems. 87	76

3.2.1. Hard systems approach

Nineteen reviews used a hard systems approach in which systems modeling approaches and computer simulations were used to construct models. The methods used were systems dynamics (n = 16), agent‐based modeling (n = 1), social network analysis (n = 1) and “cross‐over” modeling technique (n = 1).

Systems dynamics

Sixteen included reviews used systems dynamics to build models to represent the dynamic complexity of a system. ⁵⁰ , ⁵¹ , ⁵² , ⁵³ , ⁵⁴ , ⁵⁵ , ⁵⁶ , ⁵⁷ , ⁵⁸ , ⁵⁹ , ⁶⁰ , ⁶¹ , ⁶² , ⁶³ , ⁶⁴ , ⁶⁵ , ⁶⁶ , ⁶⁷ , ⁶⁸ In the included papers, the number of systems dynamics steps described to produce a systems model ranged from three to five. For most of the reviews using systems dynamics, the findings from the literature were generally useful for problem identification, hypothesis, and model formulation. In a first step, the literature was used to articulate the problem, identify the problem relevance, identify the purpose of the model, and understand the concepts to be studied.

In the second step, the literature was useful to identify the elements (often named variables, factors, components, or parameters), prioritize factors, determine the systems boundaries, and explore links between the elements. This allowed the development of an initial systems model named a causal loop diagram that maps variables and plots causal relationships between variables, and can generate dynamic hypotheses (n = 13). ⁵⁰ , ⁵¹ , ⁵² , ⁵⁶ , ⁵⁷ , ⁵⁸ , ⁵⁹ , ⁶⁰ , ⁶² , ⁶³ , ⁶⁴ , ⁶⁵ , ⁶⁷ For example, a causal loop diagram on the influence of organic farming on the growth in food production is provided in Brzezina et al. ⁵² The diagram provides information on the causal links between them through feedback loops. For instance, a feedback loop shows that maximization of profits (variable 1) can lead to reinvest in machinery (variable 2), which can increase yield (variable 3) and food production (variable 4), and thus increase profit (variable 1).

In the third step, the literature could be used to determine parameter values to be entered into simulation models. This allowed the depiction of a systems dynamics model, named a stock‐and‐flow diagram (n = 7). ⁵² , ⁵⁸ , ⁵⁹ , ⁶¹ , ⁶³ , ⁶⁴ , ⁶⁸ For example, Zammar et al. ⁶⁸ developed a systems diagram to depict the factors affecting the willingness to participate in clinical trials (flow) that regulate the amount of clinical trials (stock).

The last steps of systems dynamics involve testing, validating and calibrating the model, performing sensitivity analysis, and formulating policy implications. For these steps, other types of data collected from qualitative or quantitative methods were used, such as data from case studies, ⁵⁹ , ⁶⁰ , ⁶¹ , ⁶⁹ mixed methods studies, ⁵⁵ , ⁶⁷ and real or synthetic data. ⁵² , ⁵³ , ⁵⁴ , ⁶³

Among the 16 reviews, four provided information on the methods used to synthesize data from the literature. They used content analysis (n = 1), ⁵¹ realist synthesis (n = 1), ⁵⁸ , ⁵⁹ and meta‐analysis (n = 2). ⁵⁰ , ⁵⁴ The software used to perform systems dynamics were: Vensim (n = 9), ⁵⁰ , ⁵² , ⁵³ , ⁶¹ , ⁶² , ⁶³ , ⁶⁴ , ⁶⁷ , ⁶⁸ and AnyLogic PLE (n = 1). ⁶⁵

Two reviews using systems dynamics also employed other methods. One review, on technological and economic factors responsible for schedule and cost overrun in research and development projects, used methods for prioritizing and ranking parameters to be included in subsequent models. ⁶¹ Before developing the systems dynamics model, this review used the analytic network process ⁷⁰ to calculate a risk priority index of technological and economic factors. Analytic network process was performed using the key factors identified from the literature review and expert opinions. Then, a group of experts judged the weight of each factor and performed pairwise comparisons using the Super Decision software. This allowed identification of the factors that have the highest influence on schedule and cost overrun, which were considered when developing the causal loop diagrams.

Another review interested on skill needs for agri‐food nanotechnology incorporated nonlinear systems dynamics/mixed‐methods modeling. They used qualitative systems analysis to identify system variables and analyze their mutual interactions from data collected in a systematic review, surveys, and interviews with experts and stakeholders. ⁶⁷ The authors also performed strategic flexibility analysis to assess the capability of a system to respond to the dynamic environment through changes and actions. ⁷¹ The findings from these methods were used in a final phase to develop causal loop diagrams.

Agent‐based modeling

One paper mentioned using agent‐based modeling to understand how the formation and evolution of leisure‐time physical activity population patterns occur. ⁷² This method was used to represent the interactions and causal pathways between the agents (i.e., entities of a system) and the environment in a dynamic manner, and to explore the behavior of complex systems. This paper developed a conceptual model to inform the design of agent‐based modeling. Data from the literature review and consultation with experts were used to build a dynamic conceptual model on the psychological and environmental factors influencing leisure‐time physical activity. The next steps planned by the authors include developing and parameterizing the agent‐based model as well as validating the model with real‐world data. ⁷²

Network analysis

One paper conducted a literature review of dynamic modeling studies to identify parameters that impact the progress of construction projects. ⁷³ They used social network analysis to investigate the interconnectivity among the parameters. This was done by calculating the degree centrality of each parameter, which measures the number of connections one parameter has to the other parameters. The network diagram shows the links and the link strength between the parameters.

“Cross‐over” modeling technique

A systematic review was performed to identify theoretical elements and behaviors of asset management data infrastructure and formulated four propositions that were tested in a case study. With the results from both methods, the authors mentioned using a “cross‐over” modeling technique to develop a data model. ⁶⁹ It consists of a modeling technique that considers problems from the agent perspective and provides insight into the relationship between agents. The model was built using the Resource Description Framework (RDF), a data model that describes the semantics of information.

3.2.2. Soft system approach

Two reviews mentioned using a soft systems methodology. ⁷⁴ , ⁷⁵ In Rushton and Lindsay, ⁷⁴ a soft systems methodology was used to structure the analysis of the literature on clinical education. They adapted the soft systems methodology and followed three steps: (1) inquiry into situation and analysis, (2) further analysis, and (3) debate and proposed development or change. A content analysis was performed to inform the definition of the different subsystems (e.g., students, supervisor, and tutor) and explore the interactions between the subsystems of the system.

Another review performed a meta‐narrative synthesis of the literature of service quality in information sciences to identify the storylines of the research area. ⁷⁵ They used soft systems methodology during the synthesis, which consists of an action‐oriented methodology designed to tackle problematic and messy situations. The review used the CATWOE analysis (Customers, Actors, Transformation process, Worldview, Owners, Environmental constraints) to identify and classify the different elements in each storyline and the links between them. This allowed to develop a rich picture of the storyline of service quality research in which the relationships between the clients, actors, transformations, worldview, owners, and environment (CATWOE) are provided.

3.2.3. Other methods

Another method was identified in one review that used a system of systems framework. Dicks et al. ⁷⁶ conducted a review using an inductive grounded theory approach to identify factors influencing the in‐hospital experience of families of potential organ donors and explore relationships within and between factors. During the interpretation of their findings, they used the system of systems theoretical framework. The two systems in their review (i.e., family system and hospital staff system) were broken down into their components and interconnectedness between the component systems were explored. The reviewers developed a systemic map of the psychosocial factors related to potential organ donation using the Kumu systems mapping tool. ⁷⁶

3.3. Rationales for using a systems methods

Among the papers using systems methods, 10 different reasons for using a systems perspective were identified. The most frequent reason was that systems perspectives allowed authors to understand and account for complexity (n = 15). ⁵⁰ , ⁵¹ , ⁵² , ⁵³ , ⁵⁶ , ⁶⁰ , ⁶¹ , ⁶² , ⁶⁴ , ⁶⁵ , ⁶⁸ , ⁶⁹ , ⁷² , ⁷³ , ⁷⁴ In these reviews, they described the complexity of the problem under study and the importance of using systems perspectives to study this complexity. Another common reason was that the systems perspective recognizes the dynamic interaction and synergies between several elements and can simulate interconnected feedback (n = 13). ⁵⁰ , ⁵² , ⁵⁴ , ⁵⁶ , ⁵⁷ , ⁵⁹ , ⁶⁰ , ⁶² , ⁶⁴ , ⁶⁵ , ⁶⁷ , ⁷² , ⁷³ Other reviews justified their use of a systems perspective because it allowed authors to view the system as a whole (provided a holistic view) (n = 8). ⁵⁰ , ⁵¹ , ⁵⁷ , ⁶¹ , ⁶² , ⁶⁸ , ⁷³ , ⁷⁶ Also, some mentioned it provided a different or new way of thinking compared with the “traditional” way that has some limitations, such as an oversimplification of the problem, assuming a linear process, and not fully grasping impacts of changes (n = 2). ⁷³ , ⁷⁵ Other reasons were that systems perspectives account for variations in the degree of influence of stakeholders and unpredictability of their behaviors (n = 2), ⁵⁰ , ⁶⁹ to make predictions by studying different scenarios of behaviors under certain conditions and circumstances over time (n = 2), ⁶⁸ , ⁷³ and to help to understand and identify critical factors (n = 1). ⁶⁸ Also, one review mentioned that the systems approach allowed the reviewers to provide a richer, more nuanced, and truthful representation of a phenomenon. ⁷⁵ Finally, papers mentioned using a systems perspective to identify knowledge gaps, and to inform future work such as to guide the prioritization of solutions (n = 2), ⁵⁰ , ⁷³ and to improve decision‐making and program development and impact (n = 6). ⁵⁰ , ⁵² , ⁶⁴ , ⁶⁵ , ⁶⁷ , ⁷³

3.4. Limits and challenges of using systems methods

Besides common limits and challenges found in literature reviews such as the paucity of the literature, missing relevant studies, non‐comprehensive literature searches, and high heterogeneity among studies, the included reviews provided eight limits and challenges related to the use of systems methods. First, one challenge often discussed is about the difficulty of capturing the true nature and complexity of the problem under study (n = 7). ⁵⁰ , ⁵¹ , ⁵² , ⁶¹ , ⁶³ , ⁶⁸ , ⁷² For example, Kumar and Thakkar ⁶¹ mentioned that “it is difficult to capture the true nature of the decisions and motives for actors' decisions which are dynamic in nature and changes over a period of time.” There is thus a need to update the developed models with time. Also, reviews mentioned that the systems model is a simplified representation of real life (n = 3) ⁵³ , ⁶³ , ⁷⁶ and might be missing elements or links (n = 6). ⁵⁰ , ⁵¹ , ⁶² , ⁶⁵ , ⁷² , ⁷⁴ For example, Adamu et al. ⁵⁰ said it is likely that they have missed factors or links in the model since they relied only on published literature. Moreover, some specified that the model did not account for variations from populations, ethnicity, culture, socioeconomic status, or geographic locations/countries (n = 3) ⁵³ , ⁵⁶ , ⁷² and this can limit the generalizability of the results since they focused on a specific area or population (n = 4). ⁵⁶ , ⁶¹ , ⁷² For example, Garcia et al. ⁷² mentioned that most of the evidence found in the literature was primarily from high‐income countries, which can limit the capacity for generalization. Another limitation was that the model was theoretical (based on literature only) and needed to be empirically tested and calibrated (n = 5). ⁵¹ , ⁶⁰ , ⁶³ , ⁶⁴ , ⁶⁸ Moreover, some mentioned they did not perform weighting, adjustment, or subgroup analysis based on some characteristics of included studies (n = 4). ⁵⁰ , ⁵⁵ , ⁶⁸ , ⁷² For example, Frerichs et al. ⁵⁶ said that they reviewed studies with a range of research designs and sample sizes, and they did not rank or weigh evidence according to design strength or assess the magnitude of effect sizes. Finally, some reviews discussed the influence of the authors' experience and preferences on the choice of the approach and parameters used, and their implication on the model found (n = 3). ⁵⁰ , ⁵² , ⁶⁷

4. DISCUSSION

Systems perspectives have been growing in popularity in evidence synthesis over the past decade. ⁶ , ⁷⁷ , ⁷⁸ In this methodological review, 98 reviews adopting a systems perspective were analyzed. These reviews addressed different aims such as to conceptualize a phenomenon, identify and understand risk factors, and examine the process and outcomes of an intervention (see Files S1 and S2). Two main categories of papers were identified: reviews using a systems lens, and reviews using systems methods. The main reasons for using systems perspectives were to address complexity, view the problem as a whole, and understand the interrelationships between the elements. Adopting this way of thinking can open the doors to a variety of methods and methodologies that focus on adopting a more holistic perspective and considering the complex nature of a system. Some papers, however, discussed the challenges of capturing the true nature and complexity of a problem and potential limits of these methods used, such as the risk of missing factors and interconnections, and not accounting for population or contextual variations.

Beside from using a systems lens to guide a review, this methodological review also identified a variety of methods to analyze data. Evidence synthesis researchers might be familiar with some of the methods used such as thematic synthesis, realist synthesis, framework synthesis, and meta‐analysis. However, several systems methods were identified that may be new to evidence synthesis researchers, such as systems dynamics, agent‐based modeling, and soft systems methodology (Table 4). They enable many aspects to be considered within a system by focusing on the linkages, interrelationships, interactions, and behaviors among the elements that characterize a whole system. ⁷⁹ , ⁸⁰ Also, when using a systems perspective, the questions addressed can be different. For example, in the literature on systems thinking, researchers have asked questions such as How do the elements interact with each other to form whole systems? How do boundary distinctions define systems? and How do the different perspectives influence the interrelationships, systems, and boundary distinctions within a system? ¹⁹ Moreover, the elements studied can be different. For example, Higgins et al. ⁷⁸ have suggested a variety of synthesis possibilities to address the different attributes of complex systems in systematic reviews such as using meta‐regression to consider the interactions between the intervention and context, the system adaptivity, and the emergent properties. Thus, when adopting a systems perspective, there is a shift of focus from specific outcomes and the different components of an intervention toward the different attributes of a complex system (Table 1).

Three main types of systems approaches have been proposed in the literature (hard, soft, and critical; see Table 2). In this methodological mapping review, the majority of included reviews used a hard systems approach, two reviews mentioned using soft systems approach, and none adopted a critical systems approach. Although there are epistemological differences between these three approaches, when applied to reviews, several similarities could be found. In general, regardless of the approach used, the findings from reviews were useful to define the problem, understand concepts, identify and define key elements, investigate and understand relationships between the elements, develop an initial systems model, and/or identify research gaps. The main difference between the approaches is in the methods used and interpretation of the findings. In this methodological mapping review, a variety of quantitative and qualitative methods as well as different combinations of methods were identified for tackling complex situations. In the reviews that used a soft systems approach, ⁷⁴ , ⁷⁵ the methods were predominantly qualitative such as meta‐narrative synthesis and content analysis. In the hard systems approach, although modeling systems methods are predominantly quantitative, they can also involve qualitative methods. For instance, the first steps of systems dynamics consist of creating initial models such as causal loop diagrams and stock‐and‐flow diagrams, and are considered “qualitative” systems dynamics since they do not involve computer simulation modeling. ⁸¹ These steps can involve identifying and selecting systems variables, and performing qualitative analysis of the interaction among the variables. In the last steps of systems dynamics, quantitative methods could be more important since they consist of testing and calibrating the model using simulation modeling and performing sensitivity analysis. In addition, these steps require to go beyond the conventional systematic review, and incorporate a wider range of evidence in the synthesis.

The examples of reviews using systems methods presented often had two characteristics in common: first, the reviews contained multiple processes, often involving the development of a conceptual model, followed by more quantitative analysis. Second, the reviews utilized a wide range of types of research to inform the development of models, and the specification of parameter values. Neither characteristic is usually part of a conventional systematic review, where usually a specific question is addressed using a relatively homogenous set of included studies. Thus, it would appear that the use of a systems method is not something that can be easily added into a conventional systematic review. It rather requires detailed upfront planning which should include an appraisal of team skills and timelines. The methods described above often require expertise that is not always present in systematic review teams. Also, the multicomponent nature of the review process will probably require longer timelines than conventional systematic reviews need.

4.1. Limitations

Some limitations of this methodological review need to be acknowledged. The search was limited to papers published in English or in French found in seven databases and three search engines. Also, the search strategy included a range of terms related with “systems perspective.” A search term on “systems dynamic” was included in the search strategy to refer to one of the characteristics of complex systems (see Table 1). As seen in the results, “systems dynamic” can also refer to a method. Some individual named methods were not part of the search strategy (e.g., soft systems methodology, critical systems approach), which may mean they were less likely to be identified if they exist. Moreover, although some non‐discipline‐specific sources were searched, evidence synthesis having adopted a systems perspective in disciplines not indexed in the sources may have been missed. Finally, there was no duplicate eligibility assessment and data extraction. Another reviewer was involved in 5% of the screening and data extraction to help clarify the eligibility criteria and data to extract.

5. CONCLUSION

Using a systems perspective offers a way of understanding complex systems and can be particularly relevant when addressing complex situations. This methodological mapping review has identified a variety of methods that were used to understand complex dynamic interactions between various elements within a whole system. These methods provide different analytical possibilities from existing methods for addressing complexity when applied to evidence synthesis. Also, reviews can combine a diversity of evidence to provide a broad and more complete picture of the literature of a complex system. These approaches can enhance the applicability of findings from evidence synthesis for patients, stakeholders, and decision‐makers by taking account of context and thinking more broadly in terms of the interaction of the different elements within a system. ⁷⁷

The results of this methodological mapping review provide a first step to understand how systems perspectives can be used in evidence synthesis. It may be useful for helping to define what we mean by a “systems perspective evidence synthesis”; identifying the conditions under which it might be useful to take this approach; and giving reviewers a starting point when designing and planning a systems perspective evidence synthesis. More methodological development is still needed to provide guidance on how to address systems complexity in evidence synthesis, which systems methods are appropriate in which circumstances, and how to integrate the different data and levels of research. The challenges for using systems perspectives are numerous, requiring a change in the way one sees and thinks about the world. There is a need to develop new knowledge and skills to implement these approaches. Also, there is a need to establish collaborations between systems researchers and evidence synthesis researchers, and to reflect on what different disciplines and methodologies can learn from each other to improve the development of research on complexity.

AUTHOR CONTRIBUTIONS

QNH and JT conceived and designed the review. QNH was involved in all the steps of the review with the collaboration of CS for the search, MB for the selection and data extraction, and DK, AOM and LvG for the synthesis. All authors critically revised and approved the manuscript.

Supporting information

Appendix S1 and S2 Supporting information

Hong QN, Bangpan M, Stansfield C, et al. Using systems perspectives in evidence synthesis: A methodological mapping review. Res Syn Meth. 2022;13(6):667‐680. doi: 10.1002/jrsm.1595

DATA AVAILABILITY STATEMENT

Data sharing is not applicable to this article as no new data were created or analyzed in this study.