Childhood Obesity Evidence Base Project: A Rationale for Taxonomic versus Conventional Meta-Analysis

Abstract

Introduction: There is a great need for analytic techniques that allow for the synthesis of learning across seemingly idiosyncratic interventions.

Objectives: The primary objective of this paper is to introduce taxonomic meta-analysis and explain how it is different from conventional meta-analysis.

Results: Conventional meta-analysis has previously been used to examine the effectiveness of childhood obesity prevention interventions. However, these tend to examine narrowly defined sections of obesity prevention initiatives, and as such, do not allow the field to draw conclusions across settings, participants, or subjects. Compared with conventional meta-analysis, taxonomic meta-analysis widens the aperture of what can be examined to synthesize evidence across interventions with diverse topics, goals, research designs, and settings. A component approach is employed to examine interventions at the level of their essential features or activities to identify the concrete aspects of interventions that are used (intervention components), characteristics of the intended populations (target population or intended recipient characteristics), and facets of the environments in which they operate (contextual elements), and the relationship of these components to effect size. In addition, compared with conventional meta-analysis methods, taxonomic meta-analyses can include the results of natural experiments, policy initiatives, program implementation efforts and highly controlled experiments (as examples) regardless of the design of the report being analyzed as long as the intended outcome is the same. It also characterizes the domain of interventions that have been studied.

Conclusion: Taxonomic meta-analysis can be a powerful tool for summarizing the evidence that exists and for generating hypotheses that are worthy of more rigorous testing.

Introduction

The development of systematic reviews in the health sciences grew out of early recognition in the 1980s that many reviews of research did not meet rigorous methodological standards.¹ Cochrane introduced the term systematic review regarding medical evidence and this term began to be used for reviews that met the high methodological standards expected of original research.² Kass suggested methodological standards for reviews that paralleled those in primary medical research, while Cooper made similar suggestions for review standards in the social sciences.^3,4

While there are no universally accepted standards for developing systematic reviews, there is wide agreement on the main stages of systematic reviews.⁵ The first stage is the problem formulation stage in which the precise question is articulated. The second stage is the data collection stage, which typically involves a literature search for studies that are relevant to the question. The third stage is the extraction of including information about effects and other relevant study characteristics (e.g., information about settings). The fourth stage is data evaluation in which the quality of the studies is evaluated. And the fifth stage is data analysis in which evidence is combined or synthesized across studies.

Since its introduction, the use of meta-analysis in systematic reviews has become widespread and a wide variety of meta-analytic statistical methods have been developed.⁶ However, there is currently a need for meta-analytic techniques that allow for the synthesis of learning across seemingly idiosyncratic interventions such as those used by the National Collaborative on Childhood Obesity Research's (NCCOR) Childhood Obesity Evidence Base (COEB) project. For example, conventional meta-analysis has previously been used to examine the effectiveness of childhood obesity prevention interventions. However, these traditional conventional meta-analyses tend to examine narrowly defined characteristics and elements of obesity prevention initiatives, and as such do not allow the field to draw conclusions across settings, participants, or subjects.

While these provide valuable insights into the effectiveness of carefully delimited approaches for specific outcomes, conventional meta-analytic findings cannot be generalized across intervention components that have the same intent but do not use the same delivery methods, intended recipients, curriculum content, settings, or populations. Therefore, as a field, knowledge is accumulated in a segmented way, piece by piece.

Although the term meta-analysis technically applies only to the analytic methods used to combine or synthesize evidence across studies, it is often used (as it is here) to refer to the entire enterprise of systematic reviewing using statistical methods. The terms conventional and taxonomic meta-analysis methods are considered in this article as synonymous with the process of systematic reviews.

Conventional Meta-Analysis

Systematic reviews using meta-analysis typically begin with the formulation of the problem by specifying rules to decide which studies to include, which variables (e.g., study characteristics) to code, and statistical analyses of the components of those studies.

Contemporary guidelines (e.g., the PRISMA guidelines) for doing systematic reviews and meta-analyses recommend that these decisions be specified in a registered protocol before the data collection begins.⁷ There are different kinds of statistical analyses depending on the precise question the systematic review seeks to answer. Most often, analyses focus on the typical or average effect across studies evaluating a specific type of intervention.

Taxonomic Meta-Analysis

Taxonomic meta-analysis also begins with a problem formulation, but the formulation is much broader than that in a typical systematic review or conventional meta-analysis. The problem formulation involves describing a body of literature via systematic coding of the features of the studies. Like conventional meta-analysis it also begins with the specification of rules for including studies in the synthesis, but then proceeds to develop the coding categories for interventions, their intended recipients, and contexts. This results in a taxonomy-specific database.

Taxonomic meta-analysis widens the aperture of what can be examined and understood using meta-analysis to synthesize learnings from across a field, in this case, across the field of childhood obesity prevention. To uncover patterns across interventions with diverse topics, goals, settings, and research designs, a component approach is used in which interventions are examined at the level of their essential features or activities.^8–10 That is, rather than look at interventions as a whole, we disaggregate each to identify the concrete methods that are used (intervention components), characteristics of the intended recipients, and facets of the environments in which they operate (contextual elements). We then may find that seemingly unrelated interventions share components.

Breaking an innovation into specific components is a well-established practice in implementation science, where it allows researchers to examine if and the extent to which components occur, and how they relate to desired outcomes.¹¹ We can apply the same principle to childhood obesity prevention interventions and isolate the discrete activities that may relate to improved child weight status in a variety of contexts. Specifically, we generate taxonomies of intervention components, intended recipient characteristics, contextual elements, and desired outcomes with standardized definitions that can be applied to a range of interventions or initiatives. This is a novel application of the component approach, but an idea that has been suggested by others.^10,12,13

Thus, taxonomic meta-analysis allows both the outcomes and the intervention components to be compared across studies, which further allows researchers to examine which intervention components may be more effective for particular outcomes, rather than childhood obesity prevention or public health in general. For example, effective strategies for increasing physical activity may be different from effective strategies for behavior change in food intake, which target person-based behaviors. Use of taxonomic meta-analytic methods further allows for characterizing the context and the targeted unit or level (e.g., community vs. school system) of the implementation strategy. This information provides understanding regarding under what conditions and in what settings a particular approach is effective with intended recipients.

Taxonomies are created by discovering them from the literature itself, using a qualitative research technique called grounded theory.^14,15 For the COEB project, taxonomies with outcomes, intervention components, intended recipient characteristics, and context were derived from a training set of 40 articles. More details on the methods for creating taxonomies for taxonomic meta-analysis are described in the accompanying Methods article.¹⁶

How Taxonomic Meta-Analysis Differs from Conventional Meta-Analysis

Taxonomic meta-analysis and conventional meta-analysis differ in both procedures and objectives. Perhaps the most obvious difference is one of procedure. Conventional meta-analysis fixes a set of coding categories and analyses before coding begins, but in taxonomic meta-analysis, the codes emerge as a part of the analysis. Because the coding categories emerge as part of the analysis, there are little or no missing data on codes in taxonomic meta-analysis because the categories are only selected if there is information available to code them. In conventional meta-analyses, the object is to identify the effects of one or more types of interventions that are defined in advance. In taxonomic meta-analysis, the objective is to model the variation of effects as a function of intervention components, contexts, intended recipients, and methodological characteristics. In the case of the COEB project, the taxonomy was organized by relevant community-based prevention theory and therefore grounded by the socioecologic model.¹⁷

Conventional meta-analysis has a hypothesis-testing orientation (e. g., Does the average effect of the treatments support that the treatments are effective?). Taxonomic meta-analysis has an orientation toward hypothesis generation (e.g., Are certain patterns of treatment components associated with larger effects?). However, taxonomic meta-analyses also allow testing the effects of previously proven approaches as mediators or moderators of treatment components.

This orientation, hypothesis generation while testing the effects of known influences on the desired outcome, means that the product of the taxonomic meta-analysis is not just the report of the particular pattern of associations found, but also a data set that can be used by other researchers to discover new patterns. For the COEB project, the data set will be available online at https://www.nccor.org/childhood-obesity-evidence-base-test-of-a-novel-taxonomic-meta-analytic-method/

Furthermore, conventional meta-analyses differ from taxonomic meta-analyses at each stage of development. In the problem formulation stage, conventional meta-analyses typically study the effect of an intervention with specific theoretical components having a specific mechanism of action operationalized in a prespecified manner. Taxonomic meta-analyses typically study the effects of interventions with a range of theoretical components and different mechanisms of action, operationalized in many different approaches. At the data collection stage, conventional meta-analyses typically have inclusion rules that limit the set of admissible research designs, and therefore data, whereas taxonomic meta-analyses typically include a wider range of research designs and sources of evidence.

While it is not part of the taxonomic method, taxonomic meta-analysis is compatible with the use of strength of evidence assessments for systematic reviews using diverse sets of evidence (not just randomized controlled trials [RCTs]) such as HEALM.¹⁸ For the COEB project, the need for inclusion of diverse research designs and data inclusion was supported by input by the project External Expert Panel. With that support, however, standardized inclusion criteria were developed as discussed in the accompanying Methods article.¹⁶

At the data extraction stage, conventional meta-analyses typically focus on one prespecified effect size and a narrow set of predefined contextual characteristics. As stated in the need to be more inclusive of evidence generated by diverse initiatives, taxonomic meta-analyses typically focus on a wider range of design, contextual, and program component characteristics and a wider range of study effect indicators. At the data evaluation stage, conventional meta-analyses typically focus on a narrow range of design types or study quality ratings intended to assure internal validity (freedom from bias).

Although taxonomic meta-analyses typically include a wider range of design types, similar to traditional meta-analysis, coded characteristics potentially related to internal validity (freedom from bias) or external validity (generalizability) are included. In the data analysis stage, conventional meta-analyses typically focus on average effects and their consistency across studies. Taxonomic meta-analyses focus on modeling variation as a function of components of interventions, characteristics of the intended recipients, contexts, and methodological characteristics of the study.

Finding Patterns in the Studies

Many analytic strategies can be used to find patterns in a database of studies prepared for taxonomic meta-analysis. The simplest probable use of a taxonomy-based database is to make summaries of the available studies: to generate “maps” of the available evidence and to indicate gaps where there is little or no research evidence. For example, one can explore very specific questions, “Are there any experimental studies of stepped intensity interventions for one type of population? And if so, what effects did they find?” Alternatively, one might explore more general questions such as “What interventions have been studied for one type of population relative to a disease outcome?” The result of such a search of the database can be arranged in various ways, but one helpful format is an evidence map that lists the relevant studies as rows, and selected outcomes, intervention components, intended recipient(s), or intervention contexts as columns. This is how the COEB project data set is structured (available online).

Analyzing Subgroup Differences and Meta-Regression

For use in taxonomic meta-analyses, the analytic strategy that most resembles conventional meta-analysis is the use of analysis of variance or regression techniques (called meta-regression in the health sciences) for effect sizes.⁶ Because analysis of variance models can be conceived and analyzed as meta-regression models, we confine our discussion here to meta-regression, which was used in the COEB project.

The statistical methods used in meta-regression differ from those used in conventional data analysis in two ways. First, use employ weighted computations that take-into-account that the sampling uncertainty of the effect estimates differs (usually very substantially) across studies. Second, they use a random coefficient model to quantify the variation not accounted for by the analytic model.

Meta-regression is used in conventional meta-analysis to examine the relationship of prespecified sets of study-level (or occasionally outcome-level) predictors with effect sizes. This kind of examination of relationships of prespecified predictors with effects can also be done in taxonomic meta-analysis and can be quite effective in investigating highly focused aspects of the taxonomy-based database.

An alternative approach that is frequently used is empirical model building. There are many variations of this approach, but one of them is forward stepwise meta-regression, which is adopted in the COEB project. It builds an analytic model by evaluating, at each step, how much each additional potential predictor would improve the model, choosing the one that improves prediction the most at each step, and continuing until the addition of further variables produces negligible improvement. The result is a predictive model that identifies which predictors (e.g., intervention components) are most related to effect and the regression coefficients indicate how strongly they are related to effects. A priori hypothesized mediators and moderators can also be tested in such models, helping to better understand which elements are more effective under what circumstances.

Coping with Situations in Which Effect Sizes Cannot Be Calculated

One of the weaknesses of conventional meta-regression is that it can only be applied to situations in which all outcomes can be represented as conventional effect sizes (e.g., standardized mean differences or odds ratios). An analysis strategy that can also incorporate outcomes for which only the direction of effect is known is available and suitable for use in taxonomic meta-analysis. Hedges and Olkin note that the signs or directions of the effects, along with sample sizes, contain information about the underlying effect size parameters.¹⁹ They show that models similar to the so-called item response models in psychometrics can be used to estimate the underlying effect size (and an uncertainty of that estimate can be calculated) from only the signs of effects in a series of studies. This approach allows inclusion of data from studies in which an a priori effect size was not defined or calculated.

Furthermore, these methods can be used to combine analyses of some studies that provide effect size estimates for their outcomes and others that provide only signs, using maximum likelihood estimation procedures that are used in meta-regression. Using maximum likelihood estimation meta-regression in this way (in fact, a statistically optimal way) provides a means to combine the information from studies that provide only directions of effects with those that provide quantitative effect size data.

Identifying Effective Interventions

The taxonomy-based database (coded elements of the intervention components, recipient characteristics, implementers, and context) should be useful for identifying effective interventions. However, the question of which interventions are effective is often too broad to be utilized for future efforts. The database developed for the COEB project, that is, specific to the issue of childhood obesity prevention, will be useful in addressing more targeted questions such as “For which level of the social ecological model are there effective intervention strategies” or “for which intended recipients or which setting do we have effective intervention strategies?”

Designing New Interventions

Another application of the taxonomy-based database is to aid in the design of new interventions. By providing evidence about which intervention components are associated with effective interventions, for which outcomes, in which populations, the taxonomic meta-analysis can suggest which components might combine to produce effective interventions in particular contexts, for particular intended recipients. The important feature of the database is that it goes beyond generic effectiveness overall and provides insight into the effectiveness of intervention components in a context- and outcome-specific manner that is lacking from conventional meta-analytic summaries in the obesity prevention literature.

Limitations and Strengths

The most important limitation of the taxonomic approach is that it does not provide rigorous tests of a priori hypotheses (as in randomized trials) regarding whether an approach is efficacious when implemented in a specified manner. Individual primary studies, such as randomized-controlled trials, are designed to generate strong causal evidence. Conventional meta-analyses that aggregate the results of randomized trials can also generate strong causal evidence, but the conceptual warrant for the causal claims rests on the strength of the causal estimates in the individual studies.

Taxonomic meta-analysis uses comparisons among studies to generate information (e.g., interventions with these features produce larger effects than interventions with those features) about the potential of using various intervention components in context. Even if the studies themselves are randomized trials, the claim that some interventions produce larger effects because they have certain identified features is a claim based on what is essentially a correlational study, with all the inherent weaknesses of correlational studies.

The distinction between aggregates of study main effects vs. comparing study effects and attributing the cause of the difference to an observed difference in studies has long been recognized in systematic reviewing and has been called the difference between “study-generated evidence” and “review-generated evidence.”⁴ While statistical methods may be used to control for differences among studies (other features that might confound estimates of a particular difference of interest), statistical control is a much weaker method of controlling confounding than exact matching or randomization. As in other observational studies, more than one explanatory model may fit the data equally well. However, like other kinds of observational studies, taxonomic meta-analysis can be a powerful tool for summarizing the evidence that exists and for generating hypotheses that are worthy of more rigorous testing using randomized trials or other rigorous methods.

Conclusions

Taxonomic meta-analysis adds to our tools to evaluate evidence derived from diverse and seemingly unrelated efforts to achieve common health goals and in this case prevention of childhood obesity. It can be an important tool in exploiting the expanding and diverse evidence base. This need has been identified through various efforts: dissemination and implementation science, systems modeling, uncontrolled but important community efforts to impact health outcomes, and policy implementation. Furthermore, it has not accounted for or examined the multiple levels of drivers of change in community-based obesity prevention efforts. It is hoped that the COEB project and the subsequent articles detailing its Methods,¹⁶ Results,²⁰ and Building Translational Capacity²¹ will provide a new perspective and method to inform future research, policy development, and community-based activities to improve obesity prevention efforts.

Funding Information

The research reported in this article was supported by the Office of Behavioral and Social Sciences Research of the National Institutes of Health (NIH) under contract number GS-00F-0007M. The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the NIH or the Centers for Disease Control and Prevention (CDC).

Author Disclosure Statement

No competing financial interests exist.