Neutrality, niches, and dispersal in a temperate forest understory

Abstract

A fundamental goal of ecology is to understand what controls the distribution and abundance of species. Both environmental niches and trade-offs among species in dispersal and competitive ability have traditionally been cited as determinants of plant community composition. More recently, neutral models have shown that communities of species with identical life-history characteristics and no adaptation to environmental niches can form spatial distribution patterns similar to those found in nature, so long as the species have a limited dispersal distance. If there is a strong correlation between geographic distance and change in environmental conditions, however, such spatial patterns can arise through either neutral or niche-based processes. To test these competing theories, we developed a sampling design that decoupled distance and environment in the understory plant communities of an old-growth, temperate forest. We found strong evidence of niche-structuring but almost no support for neutral predictions. Dispersal limitation acted in conjunction with environmental gradients to determine species' distributions, and both functional and phylogenetic constraints appear to contribute to the niche differentiation that structures community assembly. Our results indicate that testing a neutral hypothesis without accounting for environmental gradients will at best cause unexplained variation in plant distributions and may well provide misleading support for neutrality because of a correlation between geographic distance and environment.

Neutral theory (1-3) demonstrates that the compounded effects of dispersal limitation, speciation, and the role of chance through time can cause ecologically identical species to form patterns of distribution and abundance similar to those found in nature. The assumption of ecological equivalence among species in neutral theory challenges contemporary views on the importance of evolution and ecological adaptation in determining patterns of distribution and abundance, and thus current approaches to species conservation (2, 4). Although some assumptions of neutral theory have been questioned (5, 6), the value of the theory lies in the degree to which it can predict or refute the importance of species-specific traits in determining distribution and abundance. Aspects of neutral theory can be tested by model fitting (6), but a stronger test lies in directly assessing the degree to which species distributions are explained by environmental heterogeneity versus neutral processes operating on natural landscapes (4, 6, 7).

Because of the spatial effects of dispersal limitation, neutral theory predicts that the compositional similarity between plant communities will decrease as the distance between two points increases (3, 8). In contrast, niche theory predicts that community composition will change as a result of species-specific differences in evolved adaptive responses along environmental gradients (9, 10). These hypotheses are hard to distinguish in natural ecosystems, because a change in environmental conditions is often strongly correlated to a change in geographic distance (11, 12). Previous studies have attempted to partition the change in community composition attributable to environment from that attributable to geographic distance (13, 14). These studies indicate that the changes in plant distributions attributable to either of the factors independently tends to be smaller than that correlated to both distance and environment together (13), although the independent influence of environment has been shown to be more important at large geographic scales when very diverse habitats are considered (14). In other words, at large spatial scales, adaptation to specific, regional environmental regimes provides support for a niche-based perspective, but at small and intermediate spatial scales the covariance of environment and geographic distance has made it more challenging to distinguish the effects of niche versus neutral processes on community composition (7, 13). To circumvent this problem, we developed a sampling design that decouples distance and environment in a locality that is comparable in extent to previous studies with a high distance-to-environment covariance (7, 13). We thus provide a more focused test of neutral versus niche-based expectations at small and intermediate spatial scales.

Our first goal was to evaluate a neutral model of dispersal limitation as the sole determinant of species distributions. Because the neutral model assumes that species do not have specific environmental niches (1, 2), any environmental factors that covary with geographic distance could have provided misleading support for neutral spatial processes in previous studies (7, 15). Our sampling design eliminates this possibility. Our second goal was to test the relative importance of dispersal limitation and niche partitioning in determining plant distributions, and to compare the importance of these factors among different functional and phylogenetic groups. This goal differs from the first in that we relax the assumption of neutral dispersal irrespective of the underlying environment by explicitly including the effects of the environment on plant distributions in the analysis. We further assume that dispersal processes working in conjunction with environmental gradients may cause spatial patterns not predicted by a neutral model. Our third goal was to determine to what degree functionally and phylogenetically distinct plant groups respond differently to diverse environmental factors. Recognizing such differences would allow us to better understand the role of evolutionary adaptation in linking environmental heterogeneity and diversity within and among specific plant groups.

Methods

Study Site and Data Collection. Our study was based near Montreal in the Gault Nature Reserve (www.mcgill.ca/gault), an old-growth forest surrounded by agricultural and suburban developments. This rugged hill complex is considered one habitat type at a landscape scale; Acer saccharum, Fagus grandifolia, and Quercus rubra comprise 74% of the forest tree cover. The reserve contains ≈700 of the 1,600 regional species of vascular forest plants in an area of 10 km².

We used a digital elevation model of the reserve in concert with geographic information system (GIS) software to identify potential sample sites in broadly defined environmental classes based on terrain attributes (aspect, slope steepness, and slope position) (16). We took these physiographic variables as indicators for variation in a more comprehensive set of environmental variables, which were to be sampled later. In deciding potential sampling sites, we excluded all water bodies and any sites situated within 20 m of trails, the shore of Lac Hertel inside the reserve, or the outer perimeter of the reserve. We selected sites so that any correlation between environment and distance would be avoided in the data set. For example, a south-facing, steep, midslope site was chosen so as to have both similar and dissimilar sites evenly distributed across near to far distances. We also chose sites that ensured the reserve was spatially well represented (no spatial gaps) and that points had unequal distances between them, as required to maximize detection of spatial patterns (17) (Fig. 1). We iteratively tested tentative sampling designs by using Mantel tests until there was no detectable correlation between distance and site characteristics. We also assessed the selected sites in the field in spring 2002 and used initial on-site estimates of slope, aspect, soil moisture, and humus richness to again test the success of our sampling design.

Fig. 1.

Spatial distribution of sampling points. Contours are at 10-m intervals, for a total elevational gradient of 230 m over the area sampled.

Geographic distances between sampling points ranged from 0.135 to 3.515 km (Fig. 1), with the lower limit determined by the minimum distance at which a decoupling of topographic position and geographic distance could occur in this locality. The upper limit is representative of reserve sizes in this region (18). GIS analyses of the total cover for dominant tree species as well as the proportion of each forest cover type derived from aerial photo interpretation indicated that our sample was representative of forest cover in the reserve. Our final sampling design reduced the correlation between environmental difference and geographic distance to trivial levels; once a Bonferroni correction was applied for multiple comparisons, only nitrate showed a significant but minor correlation with distance (Mantel r = 0.14).

At each of the 69 sampling sites we established a 50-m² circular plot and scored percent cover for all vascular plants <1.5 m tall from May 18 to July 19, 2002, with additional sampling in late summer and fall to confirm some species identities. The 1.5-m height limit restricts analysis of tree distributions to the regeneration stage, at which point individuals are in direct competition with the understory vegetation. Therefore, for trees we test neutral and niche predictions only at the juvenile stage. Some ephemeral, spring flowering species (e.g., Claytonia and Dicentra spp.) that could not be reliably assessed across all sites were eliminated from analysis, as were a few congeners too difficult to distinguish as juveniles. To meet the assumptions of neutral theory, we also removed parasitic species (e.g., Epifagus and Monotropa spp.) from analysis.

We took three soil moisture measurements at each sampling point by using a theta probe (ML2x, Delta-T Devices, Cambridge, England) to a depth of 5 cm in early September, 2002. We pooled four soil samples from each site and analyzed loss on ignition at 500°C (as an estimate of soil organic matter), pH, , , P, K, Ca, and Mg. At each site we measured slope, aspect, soil humus profiles, the intensity of grazing and browsing, and 22 physiographic characteristics (e.g., micro- and mesoslope position, mounding, percentage of rock cover, etc.). We performed a principal components analysis (PCA) on the 22 physiographic characteristics to reduce the number of explanatory variables, and included the first five PCA axes (explaining 64% of the variance in these characteristics) as composite measures of the physiographic regime. Similarly, we used the first two axes of another PCA of direct, indirect, and total growing season irradiance to characterize light regime. In total, our environmental matrix consisted of 23 variables, many previously reported as important controls on plant distribution. Further details on sampling methods can be found in the supporting information, which is published on the PNAS web site.

Statistical Tests. In general, our statistical analyses were first performed on all plants analyzed together and subsequently on seven selected subsets of species that had similar within-group dispersal abilities. Three of these groups, which are categorized by functional form or phylogeny, have figured in previous studies of neutrality (1, 7, 8).

We first tested directly for a decrease in species similarity over distance by using three different measures of geographic distance (linear, log transformed, and rank transformed) in Mantel matrix correlations (15). The neutral expectation of the slope of the relationship between community similarity and distance will vary depending on the dispersal parameter and kernel used (i.e., the shape of the seed shadow), with a Gaussian kernel producing a linear decrease in similarity with distance on a log scale (3, 8), and a Cauchy kernel producing a monotonically decreasing curve (3). We therefore tested for a neutrally predicted decrease in species similarity with distance, with species similarity calculated as the probability of randomly drawing two individuals of the same species at each pair of sampling points (3, 8). For comparison, we also tested the effect of the environment on change in species composition by using a single Euclidean distance derived from 23 environmental variables, with this distance log transformed as necessary.

Initially, following Condit et al. (8), we planned to fit neutral curves to our results for each group. However, our analyses clearly indicated that distance did not predict species turnover for six of the groups (Table 1) and only made a minor contribution to turnover in the remaining group. Hence, we could not fit neutral model parameters to the groups because they show no neutral trend with distance. Nonetheless, to explicitly illustrate the nature of the relationship between the observed patterns and neutral patterns of distribution, we compared the distance decay of Carex species found in our plots to a series of plausible neutral expectations. We chose Carex species to illustrate the nature of the deviations from neutral expectations because they comprise a diverse but phylogenetically coherent group that are gravity- or ant-dispersed and that clearly satisfy the assumption of dispersal limitation central to neutral theory (19). We generated neutral expectations for patterns of Carex distribution by using a Gaussian dispersal kernel to model seed rain density (3, 8):

where F(r) is the probability of two individuals being of the same species at a distance r. Model values for the hypothetical curves illustrated are as follows: speciation (v) = 4.8 × 10^-8 (from ref. 8), density (p) = 1/2.66 m (from our data), and rms dispersal distance (σ) ranging from 0.25 to 5 m. Actual σ values for similar ant- and gravity-dispersed plants fall in this range (19).

Table 1. Correlations between similarity in community composition, geographic distance (neutral expectation), and environmental dissimilarity.

Plant group	No. of species	Neutral Mantel r	Environment Mantel r
Asteraceae	20	0.005	-0.235***
Carex	29	0.057	-0.233***
Poaceae	15	0.042	-0.272***
Seedless vascular plants	28	-0.003	-0.215***
Shrubs	22	0.035	-0.181***
Tree seedlings	20	-0.108*	-0.246**
All plants†	215	-0.072	-0.401***

The neutral result reports the strongest of three plausible transformations (see Methods).

^†Includes 81 vascular plant species in addition to those in the other groups listed

^*, P < 0.05

^**, P < 0.01

^***, P < 0.001

To test for species-specific niches, we used partial canonical correspondence analysis (CCA), which models each species individually and is better able to detect anisotropic spatial structuring than tests with distance matrices (15, 20). To satisfy statistical assumptions, we excluded species occurring fewer than three times, yielding a working matrix of 129 species. We began with a global analysis, which included the 129 species and 23 environmental variables. Eleven of the environmental variables were significant predictors of community composition in this global analysis (forward selection, α = 0.05), and only these were retained for separate analyses of each group. Although this approach may exclude environmental variables important to only certain groups, it offers a sound minimum estimate of the influence of the environment while protecting against the inclusion of variables that could appear significant by chance. The 11 environmental variables were soil moisture, pH, Mg, NO₃, P, loss on ignition, light (first PCA axis), north/south aspect, slope, and plot micro- and mesoslope positions (from physiographic characteristics PCA axes 1 and 2).

We followed the global analysis with a partial CCA of each group by using the 11 environmental variables together with 9 variables defining spatial relationships, which were generated from a third-order polynomial of site coordinates (15, 20). This polynomial can model more complex spatial patterns than those predicted by a neutral dispersal model; we attribute these to dispersal working within an environmentally heterogeneous landscape. All environmental and spatial variables were subject to forward selection in the partial CCA for each group. Thus, for statistical comparison, the groups had similar numbers of environmental and spatial explanatory variables (11 and 9, respectively) entered into the partial CCA. The partial CCA models four distinct components of variation in plant community composition: space-only, environment-only, space correlated to environment, and unexplained variation. Unexplained variation may be produced by unmeasured environmental variables or as an artefact of the CCA analysis (21).

Because no arch effect was present in the CCAs, we used the total of all canonical eigenvalues to determine the amount of variance explained. The CCAs were tested for significance by using permutations under a reduced model (22). Because of the partitioning methodology, only environment and dispersal components could be tested. We determined the partial contributions of environmental variables to the distribution of each plant group by removing the effects of all other significant environmental variables in each CCA. All variation explained by the environment, including that which was spatially structured, was used for this analysis.

Results

Neutral theory predicts a consistent decrease in community similarity with distance, an expectation clearly illustrated for the dispersal-limited species of Carex (Fig. 2 Upper) but in fact not observed in our plots (Fig. 2 Lower). The failure to observe the distance decay predicted by neutral models occurred not only in Carex but also in all but one of the other groups tested and in all of the plants in our plots taken together (Table 1). Only tree seedlings showed a decrease in similarity with distance, but this effect explained only one percent of the total variation in tree seedling distribution (Table 1). Once environmental effects and distance effects are decoupled, the neutral model fails to explain plant distributions on this landscape.

Fig. 2.

The distance decay of Carex predicted by neutral theory (Upper) and found in our study (Lower) (line shows a LOWESS spline, with the proportion sampled in the moving window = 0.1). Neutral model predictions are based on a number of dispersal distances found in the literature, the average density of Carex found in our study, and speciation rates found in previous studies of neutral theory (see Methods). Correlation coefficient and P values in our data (Lower) are based on a Mantel test for a negative slope with distance.

To assess alternative, niche-based models for species distribution, we used a CCA (15, 20) to identify species-specific niches as well as dispersal patterns not predicted by neutral models. All of the functional and phylogenetic groups showed strong evidence of niche partitioning and weaker dispersal patterns, with the proportion of variation in species distributions explained by the environment ranging from 2 to 20 times the variation explained by dispersal (Fig. 3). Niche partitioning occurred both within groups and also when all species were considered together. At the scale of our study (0.15-3.5 km), all plant groups with the exception of Carex showed complex spatial patterns that we interpret as dispersal acting in conjunction with environmental heterogeneity (Fig. 3). Dispersal patterns were either correlated with environmental conditions (Poaceae and tree seedlings) or separate from the environment but in patterns not predicted by neutral theory (seedless vascular plants, shrubs, and Asteraceae).

Fig. 3.

Partitioning the variation in plant distributions that can be accounted for by the environment (Env), dispersal (Dis), and environment correlated to dispersal (E-D). Note that dispersal effects here are referenced against a more complex spatial model than the simple distance decay models in neutral theory (see Methods). The thickness of arrows represents strength of relationship. Groups are the same as in Table 1, with numbers of species given in brackets.

We examined the response of each group to specific environmental factors by determining the relative importance of each environmental variable in explaining variation in species distributions (Fig. 4). In this figure, a high score on an environmental axis indicates that species within that group differentiate ecologically along that axis. The distribution of species within each group is influenced by distinctly different combinations of environmental factors. For example, seedless vascular plants showed distinct niches along a nitrate gradient without any influence of phosphorus, in direct contrast to the roles of nitrate and phosphorus in defining the niches of grasses (Poaceae).

Fig. 4.

The partial contribution of environmental variables to the distribution of each species group. Groups are the same as in Table 1, with each group standardized to total explained variation. Differences in total variation shown are due to the elimination of the covariances among environmental variables. Environmental variables are represented by their chemical symbols and by the following: H₂O, soil moisture; om, loss on ignition; wp1 and wp2, meso- and microslope positions; north, north to south aspect; light, available direct and indirect light. Rings around the origin represent percent variation, with rings increasing in 10% increments to a maximum of 30% at the outer ring.

Discussion

The significant correlation between species turnover and environmental change, and the failure to detect the distance decay expected when environment and distance are decoupled, indicate that the neutral model decisively fails to describe community composition in this old-growth forest reserve. Previous tests of neutral theory have partially or fully rejected the theory (refs. 8 and 23, respectively) but were unable to test alternate hypotheses of community composition. Likewise, a previous study in the same forest could not discriminate between niche and neutral hypotheses (7). The strength of our results indicates that both competing hypotheses must be considered directly if neutral or niche theories are to be evaluated rigorously, and that decoupling the expectations of the theories is essential to discriminating between them. Indeed, given that a previous study has shown neutral patterns in this reserve (7), our results suggest that the apparent “neutral patterns” of species aggregation may be largely due to a distance to environment correlation. Studies that have explicitly considered the importance of environmental variables or disturbance (14, 24) have arrived at similar conclusions as to the importance of habitat heterogeneity in determining species composition.

Spatial patterns modeled by the CCA indicate that some species do show dispersal patterns that occur either within the confines of, or in conjunction with, environmental gradients (Fig. 3). Changes in community composition with distance have been shown for similar groups of vascular plants at distances of up to 1 km (25) and also large distances of hundreds to thousands of kilometers (26). It appears that part of the pattern observed in previous studies is due to dispersal dynamics, as indicated by the dispersal component of our results. However, although many understory plants show strong signs of dispersal limitation (19) with only occasional long-distance dispersal (27), we found that the distributions of these species are nonetheless associated with specific environments, even within a single tract of forest. At this spatial scale and in this temperate forest, plant distributions are organized mainly by the environment, and only secondarily by dispersal events. These results are consistent with analyses of spatial genetic structuring, which indicate that realized seed flow (i.e., surviving offspring) increases as the distance to favorable environments increases (28). Dispersal limitation may be more important at very small scales (<100 m), within typical neighborhood sizes for plant populations, or at biogeographic scales where infrequent long-distance dispersal events become critical. However, recent studies have shown little spatial genetic structuring beyond very local scales (28) and strong environmental structuring of plant distributions at large scales (14), suggesting that the influence of dispersal limitation should be assessed locally. The possibility of stronger dispersal effects locally, however, would not support a neutral hypothesis in this forest, because neutrality does not provide a mechanism for niche differentiation to occur at any scale.

Niche differentiation in our forest understory community clearly occurs both within functional or phylogenetic groups and when all species are considered together. For example, the ferns Onoclea sensibilis and Matteuccia struthiopteris are both correlated to high soil moisture but differ in their responses to nitrate. Perusal of these and other species in our analysis (see supporting information) reveals environmental relationships consistent with the published literature, although the autecology of many of the plants in this forest understory is only poorly known. The general lack of well described niches for specific plant species has been used to support the assumption of species equivalence in the neutral model (7), but many of the environmental gradients to which species are responding in our study cannot be detected by casual observation. Our analyses indicate that once the underlying environment is properly quantified, the importance of species-specific niches for community assembly becomes apparent.

Although environmental factors are consistently important in determining distributions within all plant groups, it also is clear that different groups are not affected equally by the various environmental gradients (Fig. 4). Such differences among phylogenetic groups may reflect mainly evolutionary constraints, whereas partitioning within phylogenetic groups likely reflect both evolutionary and functional constraints associated with ecological segregation (29, 30). Again, contrary to neutral theory, the evolutionary history and functional ecology of a species appear to be important determinants of distribution and abundance (31). The role of functional and phylogenetic diversity in structuring plant communities across different environments deserves further study.

In conclusion, even though the spatial scale and character of our study site offers ample opportunity for dispersal on ecologically relevant time scales (19, 27, 28), neutral theory fails to explain plant distributions in this forest. We do, however, observe the contrary result: environmental determinism of plant distributions. It is clear that testing a neutral hypothesis without accounting for environmental gradients will at best cause unexplained variation in plant distributions, and may well provide misleading support for neutrality because of a correlation between geographic distance and change in environmental conditions (15, 32). Dispersal effects appear to be important, but only within the confines of environmental gradients. Future studies should focus on the interaction of dispersal and environmental adaptation in determining plant distributions (4) and recognize that different functional or phylogenetic groups may not differentiate ecologically along the same environmental gradients. The roles of environmental heterogeneity, dispersal, and evolutionary history need to be incorporated into a comprehensive theory of plant distribution and abundance if conservation efforts designed to protect entire communities are to be successful.