The Implications of Measuring Complexity

In this issue of the Annals, Katerndahl, Wood, and Jaén wade boldly into deep waters indeed: attempting to quantitatively evaluate the complexity of outpatient medical care. This work was first presented 7 years ago, and the lengthy delay in publication illustrates the conceptual and methodological difficulty of this subject. The article will be beaten up, as such pioneering work typically is, but it is not important whether this particular measure of relative complexity is right—it is important that the authors have offered a rigorous quantitative measure to debate. As Box has pointed out, “All models are wrong, but some are useful.”¹

Is this model of complexity of outpatient care useful? Since their approach is explicitly based on the assumption that they are measuring a complex nonlinear system, there are 2 issues that must be distinguished in evaluating it. One is the measurement of complexity; the other, and more important, is whether what is being measured is in fact a complex nonlinear system. The authors provide a sound grounding in information theory for their measurement, and they find also that the same rank-ordering of findings emerges when other approaches are applied to their sample. Hence, the measurement itself is reasonable.

It is one thing to calculate a statistic, however, and quite another to ask whether it means anything. Is the system being measured in fact a complex nonlinear system, one whose underlying dynamics are described by complexity science? And if so, what does that imply for measurement?

Leading thinkers in family medicine have long believed that complex systems theory can help us understand the dynamics of primary care and of primary care practices.² The footprints of complexity seem to be present: systems with rich patterns of behavior, sometimes surprising behavior, self-similarity across scales, interconnectedness, and positive-feedback interactions among agents. But while we see complexity’s footprints, we have yet to capture the beast—to prove that the mathematics of complex nonlinear dynamics describe primary care and those of traditional linear modeling approaches do not.

Not all that is complicated is complex. A system with many inputs and outputs and many functions, each of which behaves in an essentially linear fashion, can produce rich and sometimes surprising patterns of behavior and contain a great deal of information. Such a complicated system is best described by a large number of linear equations, and its behavior is theoretically fully predictable even if hard to predict in practice. A complex nonlinear system, on the other hand, is very simple in the sense that a very small number of equations model it. Yet it produces extremely intricate patterns of results, often impossible to fully describe and not predictable even in theory. The crucial difference is that a complicated system produces rich behavior by having many elements behaving linearly, while a complex system produces rich behavior from a few elements that interact in highly nonlinear fashion.

A priori, we should consider it more likely that primary care is a complex nonlinear than a complicated linear dynamic system. Experience in other sciences, from physics and meteorology to biology, has shown that in nature complex nonlinear dynamics are more common than linear dynamics, though the bulk of research has been done on the much more tractable linear systems. That experience offers us a likelihood, however, not a conclusion. More solid evidence is needed, because this is no mere academic question.

For if primary care is indeed a complex nonlinear dynamic system, measures such as the Ambulatory Diagnostic Groups³ and Ambulatory Severity Index,⁴ based as they are on linear models, are not only inaccurate—they are fundamentally wrong as measures of primary care. If the models are wrong, the results, no matter how carefully calculated or statistically significant, will be specious. Policies based on such measures will have an unintended, unexpected, and likely adverse, impact on the care of patients. What gets measured is what gets managed, and bad measures will lead to bad management—to incentive structures and other policies that overvalue individual performance metrics, or composite metrics that are simple linear combinations of individual measures. Such simple measures will improve care if the linear model is correct, but may markedly worsen overall outcomes if primary care really is a complex system.

Policies based on such linear models will also tend to disadvantage family medicine as a specialty. Linear model–derived measures will make “partialist” care appear better and completely fail to capture the overall outcomes improvement of complex nonlinear generalist care.

To avoid such inappropriate measurement, it is not enough that we assert our claim that what we do should be measured with approaches based on complex nonlinear dynamics. We must show that our use of complexity science is not merely “the emperor’s new toolkit,”⁵ that complex systems models describe important phenomena in primary care not adequately described by complicated linear models, and that measures based on complexity science deliver answers that differ from those of linear models in meaningful ways.

Demonstrating that complexity science delivers better models for primary care poses considerable methodological challenges. Inference, critically distinguishing between linear and nonlinear dynamics, is especially challenging in our setting. Definitive demonstrations of complex nonlinear dynamics in other disciplines rely on extremely large data sets, with thousands of observations of the same individual system elements over time. Work in biology comes closest to our needs in primary care systems, but much development remains before small systems can be analyzed effectively over relatively short time spans. Such approaches as agent-based modeling simulations and genetic-algorithm modeling of postulated fitness landscapes are feasible with our current tools, and they can provide at least supportive results while critical inference tools are developed.

In the meantime, measurement efforts based on the assumption of complexity are at least as reasonable as traditional linear-based measures. It is important to put them forward for consideration for 2 reasons. One reason is that we must make linear-methods measurements of primary care put up or shut up. We must put out quantified, rigorous, complexity-based measures, so that simplistic linear methods no longer get a free pass, but have to be defended—or abandoned. The other reason is that struggling to produce quantified measures based on complexity is a major step toward putting complex systems investigation in primary care on a sounder footing. The information scientist Michael Cohen has said that “to be useful, complexity research must move beyond the festival of bad metaphors.” Katerndahl, Wood, and Jaén have taken an important step in helping us make that move.