Abstract

We introduce several new resilience metrics for quantifying the resilience of critical material supply chains to disruptions and validate these metrics using the 2010 rare earth element (REE) crisis as a case study. Our method is a novel application of Event Sequence Analysis, supplemented with interviews of actors across the entire supply chain. We discuss resilience mechanisms in quantitative terms–time lags, response speeds, and maximum magnitudes–and in light of cultural differences between Japanese and European corporate practice. This quantification is crucial if resilience is ever to be taken into account in criticality assessments and a step toward determining supply and demand elasticities in the REE supply chain. We find that the REE system showed resilience mainly through substitution and increased non-Chinese primary production, with a distinct role for stockpiling. Overall, annual substitution rates reached 10% of total demand. Non-Chinese primary production ramped up at a speed of 4% of total market volume per year. The compound effect of these mechanisms was that recovery from the 2010 disruption took two years. The supply disruption did not nudge a system toward an appreciable degree of recycling. This finding has important implications for the circular economy concept, indicating that quite a long period of sustained material constraints will be necessary for a production-consumption system to naturally evolve toward a circular configuration.
Introduction
Sustainability as envisioned by industrial ecology entails industry moving away from stand-alone, once-through operation to an interconnected, complex web of interlocking industries that minimize waste and maximize reuse.1 Likewise, the circular economy concept–commonly defined as an economic system ‘in which we keep resources in use for as long as possible, extract the maximum value from them whilst in use, then recover and regenerate products and materials at the end of each service life’56–requires a massive increase of interdependency with respect to material flows. There is however an unfortunate side effect of this increasing complexity. Disruptions in one part of the system can snowball and have major and unforeseen effects elsewhere. Complexity leads to unexpected risks, especially with respect to critical materials. This “cost of interdependence” associated with increased interconnectedness can be addressed through resilience–defined as ‘the capacity of a system to absorb disturbance and reorganize while undergoing change so as to still retain essentially the same function, structure, identity, and feedbacks’.2 Resilience is essential for designing truly sustainable systems based on industrial ecology or circular economy principles.3
As summarized by Wang et al.,4 resilience is often featured in material sciences (the ability of a material to return to its original shape after deformation),5 social sciences (the ability of people to deal with significant adversity),6 and ecology (the ability of an ecosystem to bounce back from or resist disruption).7 The ecological definition of resilience underpins a great deal of work on information and communication resilience8 and supply chain and manufacturing resilience.4,9
Resilience is a natural fit for the study of material criticality, which deals with (potential) disruptions of supply chains of those materials critical to society. Sprecher et al. argued that ‘one can define the criticality of a material in terms of how resilient its supply chain is’.10 DeWulf et al. added the observation that incorporating resilience as an explicit factor in criticality screening ‘could overcome the fuzziness of having recent/current versus future-oriented elements in the criticality calculation. (...) By proposing a clear split in between criticality calculations based on recent past/current characteristics and resilience that can be based on future potential characteristics, we may come not only to a better common interpretation of criticality but also reduce the level of uncertainty in the calculation’.11
Up until now, the effects of disruptive events on critical materials have mainly been investigated through scenario modeling. Materials are judged on the ability of projected supply to meet projected demand.12−15 Current scenario modeling looks at risk factors but does not take into account how fast a supply chain can rebound from a disruption. For example, Roelich et al. performed a dynamic criticality assessment of a low carbon transition scenario, developing indicators for supply disruption potential and exposure to disruptions that change over time in response to the technological development pathway given in their scenario.16 While a useful exploration of how to quantify the disruption side of the resilience equation, they explicitly excluded factors related to how the supply chain can respond to disruptions, such as substitutability and recycling, both of which play a crucial role in resilience.
To build scenarios that take resilience into account, one needs–currently unavailable–quantitative information on how supply chains of critical materials respond to disruptions. This knowledge gap is addressed in the present work.
We introduce several novel resilience metrics specifically suited to quantifying the resilience of critical material supply chains to disruptions and validate these metrics using the 2010 rare earth element (REE) crisis as a case study. We look specifically at NdFeB rare earth magnets. This exceedingly powerful type of permanent magnet is invaluable for a quick transition to a sustainable energy system.10 China, the world’s largest producer of REEs, had long since contemplated using its dominant position in the primary production of REEs to force companies to move more of their production chains to China. This would be more profitable than exporting relatively low-value REE ore or alloys.17 While already implementing a long-term export quota, an unrelated diplomatic incident with Japan involving the Senkaku/Diaoyu islands led China to unexpectedly block the export of REEs. This caused major disruptions in the supply chains of electrical vehicles, wind turbines, and many other industries. These industries had, until then, never considered themselves at risk to these types of incidents.
In our previous work on the NdFeB system we developed a qualitative framework for assessing resilience in material supply chains. This framework consists of four primary mechanisms that promote resilience.10 On the supply side, 1) diversity of supply (e.g., primary production in different countries, recycling) is a crucial mechanism to prevent disruptions, while 2) stockpiling of materials can buffer against the impact of temporary supply disruptions. On the demand side, producers have the option of 3) improving the properties of metal alloys to reduce critical material demand, especially with respect to dysprosium content in the case of NdFeB magnets. Finally, 4) substitution can play a significant role in dampening the effects of a supply disruption, in this case study either by swapping NdFeB magnets for other types of magnets or by (temporarily) switching to a different technology that does not rely on permanent magnets. As can be seen in Figure 1, each of these four mechanisms is only available to specific actors in the NdFeB supply chain. The mechanisms influence each other primarily via the neodymium price feedback loop.
Figure 1.
Conceptual resilience model of the NdFeB supply chain (black) and the associated resilience mechanisms (green) as developed in our previous paper.10 The blue arrows indicate the direction of influence: S = same, O = opposing.
In order to quantify the speed and significance of these resilience mechanisms we analyze the NdFeB case study with a novel application of the Event Sequence Analysis (ESA) method, augmented with information from interviews. ESA is very suitable to the low data environment often encountered when investigating critical materials, as well as other noncritical minor metals. The indicators introduced in this work are used to assess the contributions of the individual resilience mechanisms to the overall resilience of a critical material supply chain, in this case study that of NdFeB. This quantification is a step toward determining the elasticities in the critical material supply chains and is crucial if resilience is ever to be taken into account in scenario work. We will also reflect on the implications of our results for the design of future circular material flows.
As Meerow and Newell write, ‘quantifying some resilience characteristics would help us expand our knowledge of the relationship between resilience and sustainability’.3 In the present study we aim to contribute to this ongoing discourse.
Review of Selected Resilience Quantification Literature
Pizzol (2015) proposes to incorporate resilience in Life Cycle Assessment (LCA) via scenario modeling.18 This is relevant work, since LCA databases include many critical materials; but no work beyond a hypothetical example has been published as of yet, and indeed Navarrete-Gutiérrez et al. suggest in an analysis of the ecoinvent database that significant changes to the basic LCA data structures need to be made before it can be used as a tool for measuring resilience.19
While not addressing resilience directly, Comtois and Slack discuss the dynamic determinants of the global iron ore supply chain, finding that there is little concern for disruption, although the possibility exists of temporary but very costly interruptions of shipping routes.20 Olson analyzes the aluminum supply chain through the lens of complexity theory and finds examples of seemingly small events having major unforeseen consequences.21
Outside the field of material criticality, resilience has been extensively studied. Quinlan et al. and Hosseini et al. provide an extensive review of resilience quantification.22,23 Here we discuss the approaches most relevant to material criticality analysis.
In the field of supply chain resilience, Pettit et al. introduce a supply chain resilience framework and quantify that in follow-up work using a survey tool to ask workshop participants to gauge the resilience in a corporate environment. This framework identifies seven categories of vulnerabilities. Resilience is then conceptualized through a set of managerial controls that counterbalance these vulnerabilities. This approach is very interesting, especially when focusing purely on the corporate level, but from the perspective of our work the method of relying on workshop participants does not produce quantitative insights of the type that we are looking for (i.e., in terms of physical units).
Indeed, literature on quantitative assessment of resilience in terms of physical units is relatively rare.24 One approach is to estimate the economic production of a system in its alternative ecological states, which is then used to calculate the benefit of having resilience against switching from a high value state to a lower value state, as i.e. done by Walker et al. for water levels in Australian farmland.25
Resilience can also be quantified by looking at existing, static indicators and investigating how these indicators would change in response to disruptions. This was for instance done by Milman et al.26 In the context of urban water systems, they developed a Water Provision Resilience indicator, which is based on an existing indicator for the percentage of the population with access to safe water. The new resilience indicator improved on the previous static indicator by adding a dynamic aspect: the odds of maintaining or improving the current level and quality of access to water over the next 50 years, despite disruptions, such as a strong population increase.
Rose introduced a dynamic economic resilience indicator that defines how regional economic output is impacted by a disruption, i.e. an electricity outage.27 Resilience is interpreted by Rose as the avoided economic damage, i.e. the maximum hypothetical damage minus the actual damage incurred by the resilient system. This makes sense when doing theoretical modeling of resilience, but finding the maximum hypothetical damage caused by a disruption is challenging in real-world case studies.
Conceptually, the resilience loss indicator introduced by Bruneau et al. is strongly related to our work. It is defined as R = ∫t0t1[100 – Q(t)]dt where R represents the loss of Q during the time period of disruption, t0–t1.28 In the work of Bruneau et al. Q would be quality of infrastructure as it is impacted by an earthquake.
In the present work, Q could represent the loss of critical material supply. However, we are not convinced that the resulting R–which would stand for percentage of disrupted production–is immediately meaningful for comparing the vastly different supply chains of critical materials. Instead, we focus on quantifying the factors that influence how fast critical material supply chains can bounce back from disruption.
Methodology
Event Sequence Analysis
Through literature research and discussions with experts we identified a number of actors for each position in the supply chain. We included in the analysis at least one actor for each supply chain position and major end-point. These actors–40 in total–were chosen because they were both key players in the supply chain and because of data availability.
We employed event sequence analysis (ESA), an empirical approach to the longitudinal study of social processes, to identify the type of actions each actor undertook and how these actions developed over time. ESA was chosen because it facilitates empirical investigation of processual phenomena, thus allowing us to make the dynamic nature of resilience mechanisms explicit. ESA enables us to study the unfolding of disruptions and resilience mechanisms as sequences of events and allows for the further qualification of events to capture information that is relevant to the development/measurement of quantitative indicators.29,30
Our application of ESA proceeds through several basic steps. First, longitudinal data on the activities of the actors relevant to our case study were collected via interviews and Internet searches (e.g., Factiva, company Web sites), mainly articles in the media, press releases, and company annual reports. These data are recorded in a data set, the entries of which are referred to as incidents. Incidents are qualitative descriptions of the activities of actors, each consisting of the following:31
1) the date at which the incident occurred,
2) description of the incident,
3) the actor(s) involved, and
4) the source of the information.
After the data are collected, the incidents are coded to make explicit what resilience mechanism each incident is indicative of.32 More specifically, by assigning codes to the incidents we parse them into events that capture different moments in the unfolding of resilience mechanisms. Chronologically ordered, our complete data set of events thus offers a rich description of the resilience mechanisms under investigation and the way in which these mechanisms are embedded in the overall process. From the characteristics of events that make up the identified mechanisms we also derive other information on interest, such as how long certain activities take (e.g., from announcing the intention to build a new mine until operational start of that mine) and which actors were active in different stages of the unfolding of the mechanisms.
In order to triangulate our results, we supplemented the event sequence analysis with general production statistics obtained directly from industry sources and through interviews with 16 experts from across the NdFeB supply chain (see the SI for more information). Figure 3 was made using Gephi.
Figure 3.
Each event in the database is tagged with a category and one or more countries. The node size indicates the frequency of events. Countries that occur in an event together are connected.
Indicators for Resilience in the Supply Chains of Critical Materials
We operationalize each of the four resilience mechanisms with the following quantitative parameters:33 1) the time lag between the disruption and the moment that a measure actually starts to have a quantifiable effect on the system (this includes both the reaction time until, for example, the decision is made to open a new mine and the time required for that new mine to come online); 2) the speed with which the mechanism can scale (i.e., learning curve, the speed with which a producer can scale up a change in production from its initial reaction to its desired final amount); and 3) the maximum magnitude of a mechanism’s effect (e.g., the maximum amount of neodymium obtained through recycling). Time lag is given in years, response speed as % of overall market volume per year, and maximum magnitude in % of overall market volume.
With respect to data quality, we are confident that the present description is an accurate description of how the sector as a whole responded to the 2010 disruption. Given the opaque nature of the REE sector and the wide diversity between actors, it would be very challenging to increase the accuracy of our results beyond the order of magnitude presented here. Since the REE price was elevated for a limited time only, this case study does not show what the response magnitude would be in case of a more permanent disruption.
Results
In the following paragraphs, we will first give overall results of the event sequence analysis and then discuss each mechanism individually and finally the system as a whole.
We tracked 220 events in our event sequence database. Figure 2 illustrates that the number of resilience events spiked in the aftermath of the 2010 crisis and again roughly two years later. We can also see that the majority of events are related to primary production and that recycling accounts for as many events as substitution and changing of material properties combined.
Figure 2.
Number of events over time, as tracked in our event sequence database.
Figure 3 shows which events, categories, and countries are most prevalent in the database and how they are connected. New primary production accounted for the majority of events and is connected to a constellation of countries where exploration for new REE mines was at some point announced. More often than not, this was also connected to Japan, a country that was very active in exploring all the resilience mechanisms.
Feedback Loops through the Price Mechanism
The price spike of NdFeB magnets in the latter half of 2010 (Figure 4) incentivized actors across the supply chain to change their behavior. The price mechanism forms the overarching feedback loop through which the supply and demand of neodymium influence each other.
Figure 4.
Herfindahl-Hirchman Index of REE primary production and NdFeB magnet production showing the extreme concentration of REE production in China prior to the 2008 crisis and its subsequent redress. The dotted line for market concentration of NdFeB production is based on industry forecasts for the 2014–2020 period.37 The purple line gives the price of neodymium in its oxide form.38
A functional supply/demand feedback loop requires the existence of a transparent market. As an indicator of the existence of such a market we suggest the ratio of material traded on the spot market (i.e., where trade is public and delivery is close to immediate) compared to the total market volume. Unfortunately, it proved to be impossible to obtain the necessary data. Furthermore, a much wider comparison with other materials would be necessary to determine at which ratio a supply/demand feedback loop would become functional. We do know that although the Chinese government has attempted to establish a spot market, the majority of REEs are still not traded in a transparent manner.34
Not only transparency of the market itself is of importance but also the transparency of companies along the supply chain. A lack of financial transparency (e.g., publication of annual reports) will hinder access to fresh capital from outside sources if companies need to expand due to a sudden increase in demand (interview Chatham House).
Diversity of Supply
Having various sources of raw material can reduce the impact of a supply disruption on the remainder of the supply chain. In the resilience framework we distinguish between primary production, recycling, and illegal mining and smuggling as sources of diversity of supply.10 While diversity of supply can be seen as a unified mechanism from a resilience point of view, there are marked differences between the actual realization of recycling and mining infrastructures. We therefore analyze these sources separately. We will not consider illegal mining and smuggling due to lack of data.
As a high-level indicator of diversity of supply, we use the Herfindahl-Hirschman index (HHI), which is equivalent to Simpson’s diversity index (D) as used in ecology:35 the market shares of relevant companies are squared and summed, providing a score between 0 and 10,000. An HHI of 10,000 indicates a single producer monopoly. In contrast, low HHI values indicate that market shares are evenly distributed among a great diversity of producers. An HHI above 2,500 is considered to be highly concentrated, indicating high market power of larger producers.36 Calculating the HHI for each step in the supply chain allows one to assess which step is most critical from a diversity of supply point of view.
Figure 4 shows that the HHI for primary REE ore production and NdFeB production have opposite tendencies.
With respect to primary production, there was an extreme market concentration in the years before the crisis, indicated by a HHI value slightly above 9,400. Chinese producers held an estimated 98% market share. In the years following the 2010 crisis the HHI index dropped to values of around 7,400.
Although the 2010 crisis led to a noticeable increase in the diversity of primary REE ore production, the opposite is the case for the production of NdFeB: diversity declined at a steady pace and is estimated to keep doing so in the foreseeable future. When looking at the data underlying the HHI calculations we see that this is caused by an increasing market share of Chinese producers, while the share of producers from other countries (mostly Japan) remains constant.
So far, we assumed that strong national policies of Japan and especially China imply that the sum of companies within each of these countries can, from a policy perspective, be analyzed as single actors. To verify this assumption, we analyzed the per-company HHI for NdFeB production (comparable data for primary production was not available). We found that globally the biggest company (Beijing Zhong Ke San Huan High-Tech Co., Ltd.) had a 13,500 tons production capacity at an estimated 65% utilization rate, giving it a 12% worldwide market share, while all the other companies had market shares of 6% or smaller.37 This results in an HHI index of ∼300, indicating that on a company level there is no market concentration and thus a single company does not have the market power to cause a significant disruption. This low HHI score on a company level but very high HHI score on a country level supports the general narrative that the 2010 disruption can be attributed to decisions made at the level of national policy, rather than by decisions made by a few dominant producing companies.
We note that there are some limitations to the data: the primary production HHI is based on overall REE production data per country.39 We make the reasonable assumption that since REEs are always coproduced together this is proportional to primary production of metallic neodymium.40 The NdFeB HHI is based on production in Japan, China, and the aggregated production of the rest of the world. This simplification does not influence the results, since the production in the rest of the world is negligible. For NdFeB production we include a forecast over the period 2015–2020.37
New Primary Production
Our research indicates that the most publicized response to the 2010 REE crisis was to build new dedicated REE mines, while the most effective response in terms of quantitative output was increasing levels of REE coproduction from existing mines.
Dozens of junior mining companies (i.e., companies that focus solely on exploration) were hopeful to be the first to supply jittery western and Japanese REE consumers with non-Chinese supply, as did a number of precrisis REE projects (e.g., Molycorp, Lynas). Their cumulative efforts can be seen in Figure 5, which clearly reflects the 2010 crisis, both the reduced Chinese output (from a high point of 130,000 tons in 2010 to 95,000 tons in 2014) and the subsequent increase in non-Chinese production (from 3,500 tons precrisis to 16,000 tons in 2014). Figure 5 shows that the time lag between the crisis and the increase in non-Chinese production was less than one year. This quick ramp up of production indicates that the production uptick was due to increased REE coproduction at existing mining projects. Our event analysis finds that the time lag between announcing the intention of starting to mine REEs and actual production is 4 to 13 years.
Figure 5.
Primary production or REEs over time, per country.
Although the relatively low current REE prices have seemingly caused most truly new REE mining projects to be put on hold, it is still plausible that new production capacity will come online somewhere in the coming decade. As it stands, the actual impact of non-Chinese REE production seems to be limited, with the maximum year-over-year increase of REE production outside China being 5,000 tons (∼4% of total production volume) for 2012–2013. Because these 5,000 tons are still significant compared to the 20,000 tons shortfall, it seems that the rate of additional primary production will slow down in the near future. This also bears out in the fact that the 2013–2014 increase is smaller than the increase for 2012–2013.
Recycling
We distinguish between two types of recycling: preconsumer recycling of material lost during magnet manufacturing (e.g., grinding losses, defects) and postconsumer recycling of NdFeB from End-of-Life equipment and products.
Preconsumer Recycling
Before the 2010 crisis, preconsumer waste was not recycled because of economic feasibility issues, although at least one trading company stockpiled the potentially recyclable material (see the stockpiling section). Preconsumer recycling is currently done via two processing routes (interview Hitachi):
1) Melting and strip casting, which can be done either at the magnet manufacturer or its supplier. For this processing route the material must be of good quality (i.e., low oxidation). This is usually the case for batches with production defects, such as cracks or insufficient magnetic strength. Only 1–2% of total production is recycled in this way.
2) Acid leaching, where the alloy elements are separated in their oxide forms. This route is used for grinding losses, which, depending on factors such as final shape of the magnet and quality of grinding equipment, accounts for 10–20% of total production.
Postconsumer Recycling
Although many lab-scale processes for recycling NdFeB have been reported,41 realistically one has to either return the material to the REE refinery stage for acid leaching or use magnet-to-magnet recycling technologies.40 The latter has the downside that the recycled material needs to be very uniform if a high grade of NdFeB is to be produced. Product design and collection infrastructure greatly influences the uniformity of recovered material and imposes practical limitations compared to the total amount of NdFeB that could theoretically be recycled. Previous research has shown that for the coming decade, NdFeB recycling is highly likely to be limited to the recovery of material from computer hard disk drives (HDDs).42
Our event analysis shows that before the 2010 price spike there was mostly academic interest in the recycling of postconsumer NdFeB. The crisis sparked a flurry of commercial activity, with press releases announcing the imminent start-up of at least seven recycling factories throughout 2011–2013. However, the actual availability of recycled NdFeB remains negligible, indicating that either there is currently no commercial scale recycling or that recycled material is sold in unpublished takeoff agreements, used internally, or does not reach the market for other reasons.
Economically viable postconsumer recycling is complicated to achieve for three main reasons: first, the inherent ease of oxidation of NdFeB makes it desirable to seal the magnets to stabilize them, which makes it more difficult to recover the magnets during the End-of-Life phase. Second, the amount of NdFeB is usually too small to warrant significant effort (e.g., manual labor) to liberate the magnet. Third, the many different grades of NdFeB (with differing chemical compositions) make it difficult to achieve high quality level recycling.
Quantification
We found little evidence of large-scale recycling in our event sequence analysis. Based on interviews we can construct the following learning curve for NdFeB recycling: the time required to go from start-up to small-scale recycling of computer hard disk drives (HDDs, 40 t/y) is 5 to 8 years. Because of difficulties associated with collecting enough HDDs it can then take another 2 to 10 years to increase that production by an order of magnitude. Figure 6 shows the upper and lower boundary of recycling, based on information that the first steps toward the recycling of NdFeB from HDDs were made in 2007. The upper boundary is based on previously estimated maximum recoverable NdFeB volumes. The maximum NdFeB production from recycled material is dependent on the lifetime of products containing NdFeB, the total production at the beginning of the lifetime of those products, and the collection rate.42
Figure 6.

Upper and lower boundaries of the potential for NdFeB recycling.
We observe that the recycling of NdFeB seems to follow the lower bound of our estimates.
Stockpiling
Stockpiles act as a buffer that can lessen the impact of temporary supply disruptions. The great variety of NdFeB alloys means that stockpiling magnets is not feasible (interview Arnold magnetics). Rather, smelters could stockpile rare earth oxides and use those to produce the specific NdFeB alloy in demand.
Timely stockpiling seems like a straightforward insurance against supply chain disruptions. Especially for producers of the final product, for whom the cost of the material is usually trivial compared to the value of the final product. However, for the smelter actor the costs of stockpiling is significant, as producing REE alloys is their core business. We have found that there is either no willingness from final product produces to provide capital to smelters for a stockpile or that the lines of communication are simply too long for coordinating such a scheme.
The act of stockpiling can in itself increase demand, thereby driving up prices. For example, Japanese car manufacturers were caught off guard by the impact of the Chinese export blockade on their manufacturing capability and started building a dysprosium stockpile at the height of the crisis. One of the largest REE traders, Santoku (Japan), had relatively large stocks of grinding waste because it had an oral agreement with its customers to recycle that material. However, it had not been economically viable to do so before the crisis, so they had stored the material, which amounted to a year’s worth of stock. During the height of the crisis, their customers demanded that Santoku would stock 2 years of supply, forcing them to buy an additional year’s worth of material at the highest cost. Later, when the cost of the material went down again, they could not recuperate this high price from their customers. Overall, this caused a loss of billions of yen for smaller players and tens of billions of yen for larger players in this part of the REE supply chain (interview Biko Chemical Company).
Quantification
Measuring stockpiling is relatively straightforward: number of months the supply chain could sustain itself while the supply is disrupted. The complicating factor in determining the optimal size of a stockpile is that the choice of which supply disruptions to target and how long these are expected to take is subjective and relies on past experience. By its very nature this cannot take into account as yet unknown types of future disruptions, i.e. ones that have not occurred before or that will occur too many steps removed in the system to be foreseeable. In sectors that have mandatory stockpiles (e.g., certain types of metals, oil) stockpile size is usually set at 1 to 3 months.
Although the indicator for stockpiling is relatively straightforward, quantification of current stockpiling levels is not. There seems to be a very large disparity across industries as well as cultures. Before the crisis, German companies had 5 weeks’ worth of NdFeB in storage, while Japanese companies had several months of supply. This reflects a difference in cultural aversion to risk, which also is a necessary perspective for understanding why some Japanese companies reacted to the Chinese embargo by increasing their stockpile to two years, while a more typical reaction of European producers was to rely on their stockpile and subsequently to use lower grade magnets or accept temporary production stops. Other factors, such as differences in the market positioning of final products may also have played a role. At the time of writing, Japanese car companies hold a stockpile of 6–12 months (interview Nissan), whereas European companies still have a stockpile of only 2–5 weeks (interview Rohstoffallianz).
Changing Material Properties
Magnet producers have responded to supply constraints by improving the properties of NdFeB, greatly reducing the required amount of dysprosium for high temperature resistant magnets. For instance, using grain boundary diffusion allows for a more precise deposition of dysprosium in the NdFeB microstructure, increasing functionality. This technology was available relatively quickly because the necessary basic research had already been performed in earlier R&D, the aim of which was to increase the maximum temperature resistance of NdFeB magnets. Although at that time the increased production costs associated with grain boundary diffusion proved to be prohibitive, the extreme price increase of dysprosium turned it into a viable proposition (interview Hitachi).
The basic R&D for grain boundary diffusion took 2 to 3 years. The subsequent scaling up of such a technology from small scale to volume production can take 6 to 24 months. As with material substitution, one also needs to take into account product life cycles: once the novel material becomes available it can take from several months to 5 years before it is actually incorporated into the final products. After three years, a reduction of up to 50% of dysprosium content had been achieved.
Material substitution
The producers of the final goods that use NdFeB can substitute on many levels, ranging from using a lower grade of the same material to outright substitution of the entire technological (sub)system (e.g., replacing wind energy with photovoltaic energy). In our previous paper10 we highlighted the two most common types of substitution:
•Material substitution: the requirement of using magnets remains in the final product design, but this requirement is met with a different material (e.g., replacing NdFeB magnets with samarium–cobalt magnets).
•Technological substitution: a product is redesigned to operate without any magnets at all (e.g., replacing a direct drive with a geared wind turbine).
On the basis of our most recent research, we add another type of substitution:
•Grade optimization: a high-performance magnet is substituted by a low performance magnet with a lower REE content. This can be done almost instantly.
The variety of substitution possibilities makes it challenging to arrive at a comprehensive quantitative indicator. Nasser et al. solved this by first collecting data on a range of indicators (substitute performance, substitute availability, comined fraction, environmental impact ratio, and net import reliance ratio), then giving these a weight, and finally calculating an overall substitutability score.43 This is an appropriate approach for comparing the substitutability of various elements for the purpose of ranking them on criticality, but it does not yield the dynamics of substitution that we are looking for in this work.
A time delay of implementing substitution can be quite significant, owing to the fact that it usually requires a product redesign. Substitution will usually occur at the end of a product life cycle, although this can be expedited in the case of acute disruptions. The delay is highly dependent on the sector. Interviewees indicated that, assuming no significant R&D is necessary, components of consumer products can be substituted within several months. Strict regulations cause the automotive industry to take a year, and the extremely risk-averse aerospace and defense sectors can take up to five years.
One major global NdFeB supplier reported that overall, ±10% of their customers substituted NdFeB for samarium–cobalt (SmCo) magnets and were not aware of any other types of material substitution among their customer base. Roughly 20% of their customers preferred to switch to lower grade NdFeB magnets. Our impression is that Japanese manufacturers tried to obtain their material at any cost, while European manufacturers sometimes opted for temporarily using much lower grades of NdFeB. Since grade optimization may negatively affect performance and/or lifetime of the final product, there is almost no publishable data available on the topic, making it difficult to estimate the upper bound effect of grade optimization on total NdFeB demand. We note that the above quoted 20% is probably an underestimation of the true extent of grade optimization.
Several audio equipment manufacturers and factory automation manufacturers reported that they almost completely substituted NdFeB magnets with non-REE magnets about 2 years after the 2010 crisis. Siemens reported in 2014 that they were working on producing wind turbines with dysprosium-free NdFeB magnets “in a few years’ time”.44 One patent described a method to replace dysprosium containing NdFeB magnets with dysprosium-free NdFeB magnets that are twice as large.45 This shows how substitution can have very different goals and effects for individual applications.
In summary, we roughly estimate that the compound effect of substitution was to reduce demand for NdFeB by 10% of total demand per year. If one is willing to implement systemic substitution, the maximum magnitude will in theory be 100%. However, interviewees indicated that for the overall market, maximum magnitude will be limited to 20–50%. Another interesting side effect of substitution is that it may disrupt other markets; the drastic price increase of NdFeB caused a knock-on price increase of ∼10% for samarium cobalt magnets.
Discussion
Four resilience mechanisms can be observed in supply chains of critical materials reacting to a disruption: on the supply side, diversity of supply and stockpiling, and on the demand side, improving material properties and substitution. We introduce a set of quantitative indicators for the assessment of these four resilience mechanisms (time lag, response speed, and maximum magnitude, see Table 1). These indicators are validated with the 2010 rare earth elements (REE) crisis as a case study, which are often considered to be among the more critical materials.46,47 We focused specifically on neodymium. By defining response speed and maximum magnitude in terms of ‘% of total market’ rather than in absolute terms, the results of this case study can be compared with other supply chain disruptions. The results are summarized in Table 1.
Table 1. Summary of Resilience Mechanism Parametersa.
| mechanism | time lag | response speed | maximum magnitude |
|---|---|---|---|
| diversity: new primary production | 1–13 years | 4% of total market/y | determined by reserves base |
| diversity: recycling | 5–8 years | <1% of total market/y | limited by production and recycle rate |
| substitution | months–5 years | 10% of total market/y | 20–50% of total market |
| changing material properties | 2–3 years R&D + 1–5 years implementation | 15% of total market/y | 50% of dysprosium content |
| stockpiling | instantaneous | instantaneous | limited by stockpile size |
Time lag denotes the lag between the 2010 REE crisis and the first observable response, with the range indicating the time it took various actors to implement a given mechanism. Response speed is expressed as the annual percentage with which the market substitutes, compared to the total market volume at the beginning of the crisis. Maximum magnitude indicates the maximum effect a resilience mechanism can eventually reach.
The most salient of the findings is that substitution was the most significant system response. This contradicts Nassar et al., who concluded, based on expert consultation, that ‘substitutions outside the REE family are often difficult to impossible’.43 Substitution is highly dependent on the specific application. Our event sequence analysis indicates that some producers rapidly adapted to increasing prices by switching NdFeB for samarium–cobalt magnets, while others temporarily used lower-grade NdFeB magnets. More thorough substitution involving product redesigns followed a year or two after the disruption. Overall, roughly 10% of total NdFeB demand was substituted each year. Our research indicates that realistically up to 20–50% of demand will be substituted, depending on future market conditions (i.e., price).
Use of dysprosium as an alloying element was reduced significantly, both by substituting dysprosium-rich NdFeB alloys for other alloys and by changing the production method of temperature-resistant NdFeB magnets.
Non-Chinese primary production also increased within a year. However, as seen in Figure 7, in absolute terms the ramp-up was smaller than that of substitution (4% of total market volume per year, compared to 10% for substitution). Since truly new (i.e., greenfield) primary production capacity takes 4 to 13 years to come online, this relatively quick increase can be attributed to increased production of REEs in mines that normally only mine other metals and for whom market conditions suddenly made coproduction of REEs worthwhile. Future greenfield primary production of neodymium will certainly feature extensive coproduction with other REEs and, depending on ore composition, with other metals as well.
Figure 7.
Chinese REE production shortfall (in red) and the combined effect of the compensating resilience mechanisms (in blue).
Significant stockpiles were available at the beginning of the disruption. However, in the perception of corporate NdFeB consumers, these stockpiles were not large enough to cover the time needed to implement measures such as substitution. This caused some actors to acquire more material at almost any cost, driving the price of REEs significantly higher than otherwise would have been justified. Thus, rather than cushioning the supply disruption by releasing material from the stockpile, additional stockpiling actually worsened the disruption into a crisis. Interviewees indicated that the current level of stockpiling is 6 to 12 months for Japanese car companies, while European companies generally have a 2- to 5-week stockpile.
Finally, recycling is of note primarily because of its trivial impact on the market, due to the problems with collecting and processing NdFeB magnets from WEEE.42
Taken together, the resilience responses were of sufficient magnitude that the supply chain should have experienced less of a price shock than it actually did, especially considering the ease of substitution and the size of stockpiles relative to the magnitude of the disruption. In the following section we turn to analyzing the supply chain as a whole to understand why this was so.
Resilience of the Overall NdFeB Supply Chain
As a whole, the supply chain was able to compensate for the 2010 disruption in less than two years. The combined effect of substitution and increasing non-Chinese production is shown in Figure 7. Two dynamics deserve to be highlighted: between 2010 and 2012 the resilience mechanisms were not able to compensate for the decrease in production. During this same period, some actors were increasing their stockpiles which led to a temporary increase in REE demand. Other actors compensated by drawing on their own stockpiles, using illegally sourced materials, or even halting production altogether. After 2012, the resilience mechanisms overshot the gap in primary production. Substitution in this period can be interpreted as so-called “demand destruction”: demand that would have occurred post-2012, had there been no disruption.
While it is debatable to what extent a two-year response time can be seen as resilient, it is reasonable to believe that, as a result of the crisis, the NdFeB supply chain system has become more diversified and resilient. However, for the system to be truly resilient, the current stockpiles should be large enough to provide resilience until the other mechanisms can take over. What this means exactly is highly dependent on the type of actor and product, but our interviewees indicated that one should generally aim for a 3- to 9-month stockpile. Increased stockpiling might be advisible, given that our research indicates that the current level of stockpiling is 6 to 12 months for Japanese car companies and European companies 2- to 5-week stockpile.
The fact that substitution and replacement of primary production, and not recycling, were the main resilience mechanisms has important implications for the idea of a circular economy. Many reports on the circular economy will implicitly or explicitly adhere to a reasoning along the lines of circularity being an easy fix for stagnating economies, resource constraints, and climate change. For example, the Ellen McArthur Foundation writes that ‘resource productivity remains hugely underexploited as a source of wealth, competitiveness and renewal’.48 The International Solid Waste Association says that ‘price signals for raw materials are a key driver in any change to the circular economy’.49 This case study provides an example where a supply disruption and subsequent price peak did not nudge a system toward circularity in any appreciable degree, and although the present work only discusses the effect of a single supply disruption, it is relevant to the overall discussion on material scarcity because of the significance and duration of this disruption. If a two-year disruption causes almost no movement toward more effective material use, this implies that quite a long period of sustained material constraints will be necessary for a production-consumption system to naturally evolve toward a circular configuration.
If not effective in nudging a transition toward circularity, the REE crisis did have a different effect. Figure 4 shows that market concentration is now higher for NdFeB production than for primary production, with production capacity increasingly being concentrated in China. This is not necessarily a problem from a supply chain disruption point of view, because technical capacity to produce NdFeB outside China exists. This statistic however shows that the Chinese government’s goal of leveraging its market dominance in REE production to force production further in the value chain to China is successful.50
Lastly, we would like to comment on the status of the NdFeB supply chain as a complex adaptive system. One of the defining elements of a complex system is that the agents inside the system act more or less blindly, which gives rise to unplanned emergent behavior. If one considers supply chains as complex adaptive systems,51 then resilience can be seen as emergent behavior.52
Rotmans and Loorbach distinguish between mechanistic emergence and reflective emergence: ‘in systems exhibiting the latter type of emergence, the observers are among the objects of the system and have some reflective capacity, which enables them to observe the emergence they produce’.53
When applying this distinction to the NdFeB supply chain, we theorize that the emergent resilience responses to the 2010 REE crisis are a form of mechanistic emergence. Many actors were only dimly aware, if at all, of what was going on with actors one or two steps removed from their own position in the supply chain. Actors reacted more or less blindly. The panic buying in late 2010 being a typical example of a resulting positive feedback loop. However, after the crisis the intense scrutiny of the entire supply chain resulted in a much higher level of information for all actors involved. When actors now decide to redesign their products to facilitate future substitution, this is an act under the umbrella of reflective emergence. Additionally, one could argue that through vertical integration of actors, both through acquisitions and takeoff agreements, the structure of the system itself has become less complex.
Future Research
Although the indicators introduced in this work are relevant to resilience analysis of critical materials in general, we chose to focus on one particular case to allow for greater depth in the exploration of the interaction between the system disruptions and resilience responses. Inevitably, by doing so we sacrifice some of the generalizability of our findings. Further study of resilience dynamics in other metal supply chains can reveal the extent to which the current results are generalizable to other supply chains. Additionally, some resilience mechanisms might unfold over periods of time that outlast the time period considered in this work. It is desirable that in the future the present case study will be revisited as well. A further development of indicators could involve a derivative type indicator, such as the rate of increase of a problem divided by the response rate.54 This dynamic aspect of systems is relevant because comparing the rate of change of the resilience mechanisms with the speed with which a system can be disrupted gives insight into the overall resilience of the system.55
Acknowledgments
We would like to thank Ibuki Komatsu for translating during the Japanese interviews and the Van Gansewinkel Groep for their financial support. This research was carried out under project number M41.5.10408 in the framework of the Research Program of the Materials innovation institute (M2i).
Supporting Information Available
The Supporting Information is available free of charge on the ACS Publications website at DOI: 10.1021/acs.est.6b05751.
The authors declare no competing financial interest.
Supplementary Material
References
- Dijkema G.; Basson L. Complexity and Industrial Ecology. J. Ind. Ecol. 2009, 13 (2), 157–164. 10.1111/j.1530-9290.2009.00124.x. [DOI] [Google Scholar]
- http://www.wrap.org.uk/about-us/about/wrap-and-circular-economy (accessed Feb 1, 2017).
- Walker B.; Holling C. S.; Carpenter S. R. Resilience, adaptability and transformability in social--ecological systems. E&S 2004, 9 (2), 5. 10.5751/ES-00650-090205. [DOI] [Google Scholar]
- Meerow S.; Newell J. P. Resilience and complexity: A bibliometric review and prospects for industrial ecology. J. Ind. Ecol. 2015, 19, 236–251. 10.1111/jiec.12252. [DOI] [Google Scholar]
- Wang J.; Muddada R. R.; Wang H.; Ding J.; Lin Y.; Liu C.; Zhang W. Toward a Resilient Holistic Supply Chain Network System: Concept, Review and Future Direction. IEEE Systems Journal 2016, 10 (2), 410–421. 10.1109/JSYST.2014.2363161. [DOI] [Google Scholar]
- Nagdi K.Rubber as Engineering Material: Guideline for Users; Hanser Verlag, 1993. [Google Scholar]
- Luthar S. S.; Cicchetti D.; Becker B. The Construct of Resilience: A Critical Evaluation and Guidelines for Future Work. Child Dev. 2000, 71 (3), 543–562. 10.1111/1467-8624.00164. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Holling C. S.Engineering Resilience versus Ecological Resilience. In Engineering Within Ecological Constraints; Schulze P. C., Ed.; National Academy of Sciences-Nat. Res. Council: Washington, DC, 1996; pp 31–43. [Google Scholar]
- Laprie J. C.From dependability to resilience; Anchorage, 2008; pp G8–G9.
- Zhang W. J.; Van Luttervelt C. A. Toward a resilient manufacturing system. CIRP Ann. 2011, 60 (1), 469–472. 10.1016/j.cirp.2011.03.041. [DOI] [Google Scholar]
- Sprecher B.; Daigo I.; Murakami S.; Kleijn R.; Vos M.; Kramer G. J. Framework for Resilience in Material Supply Chains, With a Case Study from the 2010 Rare Earth Crisis. Environ. Sci. Technol. 2015, 49 (11), 6740–6750. 10.1021/acs.est.5b00206. [DOI] [PubMed] [Google Scholar]
- Dewulf J.; Blengini G. A.; Pennington D.; Nuss P.; Nassar N. T. Criticality on the international scene: Quo vadis?. Resour. Policy 2016, 50 (C), 169–176. 10.1016/j.resourpol.2016.09.008. [DOI] [Google Scholar]
- Kleijn R.; van der Voet E.; Kramer G. J.; van Oers L.; van der Giesen C. Metal requirements of low-carbon power generation. Energy 2011, 36 (9), 5640–5648. 10.1016/j.energy.2011.07.003. [DOI] [Google Scholar]
- Alonso E.; Sherman A. M.; Wallington T. J.; Everson M. P.; Field F. R.; Roth R.; Kirchain R. E. Evaluating Rare Earth Element Availability: A Case with Revolutionary Demand from Clean Technologies. Environ. Sci. Technol. 2012, 46 (6), 3406–3414. 10.1021/es203518d. [DOI] [PubMed] [Google Scholar]
- Løvik A. N.; Restrepo E.; Müller D. B. Byproduct Metal Availability Constrained by Dynamics of Carrier Metal Cycle: The Gallium–Aluminum Example. Environ. Sci. Technol. 2016, 50, 8453–8461. 10.1021/acs.est.6b02396. [DOI] [PubMed] [Google Scholar]
- Elshkaki A.; Graedel T. E.; Ciacci L.; Reck B. K. Copper demand, supply, and associated energy use to 2050. Global Environmental Change 2016, 39, 305–315. 10.1016/j.gloenvcha.2016.06.006. [DOI] [Google Scholar]
- Roelich K.; Dawson D. A.; Purnell P.; Knoeri C.; Revell R.; Busch J.; Steinberger J. K. Assessing the dynamic material criticality of infrastructure transitions: A case of low carbon electricity. Appl. Energy 2014, 123, 378–386. 10.1016/j.apenergy.2014.01.052. [DOI] [Google Scholar]
- Hurst C.China’s Rare Earth Elements Industry: What Can the West Learn?; Institute for the Analysis of Global Security: 2010.
- Pizzol M. Life Cycle Assessment and the Resilience of Product Systems. J. Ind. Ecol. 2015, 19 (2), 296–306. 10.1111/jiec.12254. [DOI] [Google Scholar]
- Navarrete-Gutiérrez T.; Rugani B.; Pigné Y.; Marvuglia A.; Benetto E. On the Complexity of Life Cycle Inventory Networks: Role of Life Cycle Processes with Network Analysis. J. Ind. Ecol. 2016, 20 (5), 1094–1107. 10.1111/jiec.12338. [DOI] [Google Scholar]
- Comtois C.; Slack B.. Dynamic Determinants in Global Iron Ore Supply Chain; CIRRELT: 2016.
- Olson D. L. Systems perspective of the primary aluminum supply chain: Unintended consequences of participant policies. HSM 2015, 34 (4), 237–248. 10.3233/HSM-130851. [DOI] [Google Scholar]
- Quinlan A. E.; Blázquez M. B. Measuring and assessing resilience: broadening understanding through multiple disciplinary perspectives. J. Appl. Ecol. 2016, 53 (3), 677–687. 10.1111/1365-2664.12550. [DOI] [Google Scholar]
- Hosseini S.; Barker K.; Ramirez-Marquez J. E. A review of definitions and measures of system resilience. Reliability Engineering and System Safety 2016, 145 (C), 47–61. 10.1016/j.ress.2015.08.006. [DOI] [Google Scholar]
- Xu L.; Marinova D.; Guo X. Resilience thinking: a renewed system approach for sustainability science. Sustain Sci. 2015, 10 (1), 123–138. 10.1007/s11625-014-0274-4. [DOI] [Google Scholar]
- Walker B.; Pearson L.; Harris M.; Mäler K.-G.; Li C.-Z.; Biggs R.; Baynes T. Incorporating Resilience in the Assessment of Inclusive Wealth: An Example from South East Australia. Environmental and Resource Economics 2010, 45 (2), 183–202. 10.1007/s10640-009-9311-7. [DOI] [Google Scholar]
- Milman A.; Short A. Incorporating resilience into sustainability indicators: An example for the urban water sector. Global Environmental Change 2008, 18 (4), 758–767. 10.1016/j.gloenvcha.2008.08.002. [DOI] [Google Scholar]
- Rose A. Economic resilience to natural and man-made disasters: Multidisciplinary origins and contextual dimensions. Environmental Hazards 2007, 7 (4), 383–398. 10.1016/j.envhaz.2007.10.001. [DOI] [Google Scholar]
- Bruneau M.; Chang S. E.; Eguchi R. T.; Lee G. C.; O’Rourke T. D.; Reinhorn A. M.; Shinozuka M.; Tierney K.; Wallace W. A.; Winterfeldt; von D. A Framework to Quantitatively Assess and Enhance the Seismic Resilience of Communities. Earthquake Spectra 2003, 19 (4), 733–752. 10.1193/1.1623497. [DOI] [Google Scholar]
- Boons F.; Spekkink W.; Jiao W. A Process Perspective on Industrial Symbiosis: Theory, Methodology, and Application. J. Ind. Ecol. 2014, 18 (3), 341–355. 10.1111/jiec.12116. [DOI] [Google Scholar]
- Spekkink W. A. H.; Boons F. A. A. The Emergence of Collaborations. JOPART 2016, 26 (4), 613–630. 10.1093/jopart/muv030. [DOI] [Google Scholar]
- Poole M. S.; Van de Ven A. H.; Dooley K.; Holmes M. E.. Organizational Change and Innovation Processes: Theory and Methods for Research; Oxford University Press: New York, 2000. [Google Scholar]
- Sminia H.Contextualized explanation in strategy research. In Research Methods for Strategic Management; Circella G., Cinici M. C., Eds.; Routledge: New York, 2015; pp 78–97. [Google Scholar]
- Monge P. R. Theoretical and Analytical Issues in Studying Organizational Processes. Organization Science 1990, 1 (4), 406–430. 10.1287/orsc.1.4.406. [DOI] [Google Scholar]
- Rare earth products exchange in good operation: experts. Xinhua News Agency: October 7, 2014.
- Simpson E. H. Measurement of diversity. Nature 1949, 163, 688. 10.1038/163688a0. [DOI] [Google Scholar]
- http://www.justice.gov/atr/herfindahl-hirschman-index (accessed Feb 1, 2017).
- Rare Earth Market Outlook: Supply, Demand and Pricing from 2014–2020; Adamas Intelligence: 2014.
- http://www.arultd.com/rare-earths/pricing.html (accessed Feb 1, 2017).
- Gambogi J.Mineral Commodity Summaries 2015; USGS: 2015 10.3133/70140094. [DOI] [Google Scholar]
- Sprecher B.; Xiao Y.; Walton A.; Speight J.; Harris R.; Kleijn R.; Visser G.; Kramer G. J. Life Cycle Inventory of the Production of Rare Earths and the Subsequent Production of NdFeB Rare Earth Permanent Magnets. Environ. Sci. Technol. 2014, 48 (7), 3951–3958. 10.1021/es404596q. [DOI] [PubMed] [Google Scholar]
- Binnemans K.; Jones P. T.; Blanpain B.; Van Gerven T.; Yang Y.; Walton A.; Buchert M. Recycling of rare earths: a critical review. J. Cleaner Prod. 2013, 51, 1–22. 10.1016/j.jclepro.2012.12.037. [DOI] [Google Scholar]
- Sprecher B.; Kleijn R.; Kramer G. J. Recycling Potential of Neodymium: The Case of Computer Hard Disk Drives. Environ. Sci. Technol. 2014, 48 (16), 9506–9513. 10.1021/es501572z. [DOI] [PubMed] [Google Scholar]
- Nassar N. T.; Du X.; Graedel T. E. Criticality of the Rare Earth Elements. J. Ind. Ecol. 2015, 19 (6), 1044–1054. 10.1111/jiec.12237. [DOI] [Google Scholar]
- Permanent magnets are key, says Siemens wind power expert. Electronics Weekly. May 28, 2014.
- ND-FE-B Permanent Magnet without Dysprosium, Rotor Assembly, Electromechanical Transducer, Wind Turbine. April 24, 2014; pp 1–8.
- EU. Critical raw materials for the EU: The Adhoc Working group on defining critical raw materials; 2010.
- Critical Materials Strategy; U.S. Department of Energy: 2011.
- Growth Within: a circular economy vision for a competitive Europe; Ellen Macarthur Foundation: 2015; pp 1–98.
- Circular Economy: Trends and Emerging Ideas; Report 1; International Solid Waste Association, 2015; pp 1–48.
- Schlinkert D.; van den Boogaart K. G. The development of the market for rare earth elements: Insights from economic theory. Resour. Policy 2015, 46, 272–280. 10.1016/j.resourpol.2015.10.010. [DOI] [Google Scholar]
- Choi T.; Dooley K.; Rungtusanatham M. Supply networks and complex adaptive systems: control versus emergence. Journal of Operations Management 2001, 19 (3), 351–366. 10.1016/S0272-6963(00)00068-1. [DOI] [Google Scholar]
- Gunderson L. H.; Holling C. S.. Panarchy: understanding transformations in systems of humans and nature; Island Press: 2002. [Google Scholar]
- Rotmans J.; Loorbach D. Complexity and Transition Management. J. Ind. Ecol. 2009, 13 (2), 184–196. 10.1111/j.1530-9290.2009.00116.x. [DOI] [Google Scholar]
- Meadows D. H.Indicators and information systems for sustainable development; The Sustainability Institute: Hartland Four Corners VT: 1998. [Google Scholar]
- Meadows D.Thinking in systems: A primer; Chelsea Green Publishing: 2008. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.






