Skip to main content
EPA Author Manuscripts logoLink to EPA Author Manuscripts
. Author manuscript; available in PMC: 2021 Aug 23.
Published in final edited form as: Environ Entomol. 2019 Feb 13;48(1):4–11. doi: 10.1093/ee/nvy103

Workshop on Pesticide Exposure Assessment Paradigm for non-Apis Bees: Foundation and Summaries

Natalie K Boyle 1, Theresa L Pitts-Singer 1, John Abbott 2, Anne Alix 3, Diana L Cox-Foster 1, Silvia Hinarejos 4, David M Lehmann 5, Lora Morandin 6, Bridget O’Neill 7, Nigel E Raine 8, Rajwinder Singh 9, Helen M Thompson 10, Neal M Williams 11, Thomas Steeger 12
PMCID: PMC8381227  NIHMSID: NIHMS1728981  PMID: 30508116

Abstract

Current pesticide risk assessment practices use the honey bee, Apis mellifera L., as a surrogate to characterize the likelihood of chemical exposure of a candidate pesticide for all bee species. Bees make up a diverse insect group that provides critical pollination services to both managed and wild ecosystems. Accordingly, they display a diversity of behaviors and vary greatly in their lifestyles and phenologies, such as their timing of emergence, degree of sociality and foraging and nesting behaviors. Some of these factors may lead to disparate or variable routes of exposure when compared to honey bees. For those that possess life histories that are distinct from A. mellifera, further risk assessments may be warranted.

In January 2017, forty bee researchers, representative of regulatory agencies, academia and agrochemical industries, gathered to discuss the current state of science on pesticide exposure to non-Apis bees and to determine how well honey bee exposure estimates, implemented by different regulatory agencies, may be protective for non-Apis bees. Workshop participants determined that although current risk assessment procedures for honey bees are largely conservative, several routes of exposure are unique to non-Apis bees and warranted further investigation. In this forum article, we discuss these key routes of exposure relevant to non-Apis bees and identify important research gaps that can help inform future bee risk assessment decisions.

Keywords: risk assessment, pesticides, pollinators, routes of exposure

Introduction

Undoubtedly, pollinators play an important role in providing pollination services to flowering plants in both managed and unmanaged landscapes. Although honey bees (Apis spp.; Hymenoptera: Apidae) are heavily relied upon to meet a significant proportion of global crop pollination demands, wild and managed non-Apis bees, including bumble bees (Bombus spp.; Hymenoptera: Apidae), stingless bees (Tribe: Meliponini; Hymenoptera: Apidae) and solitary bees, also contribute substantially to these ecosystem services (Winfree et al., 2007, Garibaldi et al., 2013). Current risk assessment methods that evaluate the potential for adverse effects on bees resulting from pesticide exposure use honey bees (Apis mellifera) as a surrogate for all other bee species. However, non-Apis bee species differ from honey bees, and from each other, in their seasonal timing of emergence, lifespan, degree of sociality and foraging and nesting behavior. Variation in these factors could result in significantly different actual and relative contributions of routes of pesticide exposure, depending upon the species in question.

There are many advantages of using honey bees to evaluate exposure routes. They are readily available, genetically homogeneous, multivoltine, relatively easy to rear and maintain, prolific, and amenable to confined conditions (e.g., laboratory settings and semi-field studies). In addition, their oral exposure to pesticide residues are fairly easily manipulated in lab-based assessments, and methods have been developed for larval and colony responses. Many of these advantages do not apply to other commercially available pollinator species (Cane and Pitts-Singer, 2011; Peterson and Artz, 2014;).

Although honey bees may be convenient for evaluating pesticide risks to pollinators, other bee species present differential exposure to pesticides based on many factors, including, but not limited to, body size, food consumption rate, nesting habitat, nesting substrates, overwintering strategy, foraging time and behavior, and ratios of pollen and nectar in larval and adult diets.

In January 2017, 40 scientists from various regulatory agencies, academic institutions, and agrochemical industries gathered in Washington D.C. to discuss all potential routes of chemical exposure to non-Apis bees, and how well these exposure routes are addressed by current risk assessment methods for honey bees. This workshop stemmed from a series of informal discussions between stakeholders in industry, academia and government over time and in response to the conclusions drawn by a 2012 FIFRA Scientific Advisory Panel (SAP 2012), which noted that non-Apis species may be exposed to environmental pesticides differently than honey bees. Many of the participants initially met at the 2016 International Pollinator Symposium (State College, PA), which ultimately motivated the organization of this workshop. The workshop was proposed in the context of the important regulatory work of the United States Environmental Protection Agency (US EPA), Health Canada Pest Management Regulatory Agency (PMRA) and California Department of Pesticide Regulation (CAL DPR) for developing the Guidance for Assessing Pesticide Risks to Bees (EPA 2014), and the work by the European Food Safety Authority (EFSA 2013), Australia Pesticides and Veterinary Medicines Authority (APVMA 2015), and the Brazilian Institute of the Environment and Renewable Natural Resources Manual de Avaliação de Risco Ambiental de Agrotóxicos para Albehas (Cham et al., 2017) in developing guidance documents on assessing risks to bees.

More than 10 countries were represented at this meeting. Lively and lengthy discussions of different routes of exposure to solitary bees (including Nomia melanderi Cockerell; Hymenoptera: Halictidae; Megachile rotundata Fabricius and Osmia spp.; Hymenoptera: Megachilidae), bumble bees (Bombus spp.), and stingless bees (Tribe: Meliponini) led to an improved understanding of the needs and purposes of possible revisions to current exposure assessment methods and innovating new techniques/methodologies for assessment. The focus of the workshop was to assess exposure routes and not hazard (lethal or sub-lethal/behavioral effects), as participants decided that hazard assessment warranted a separate discussion. The objectives of the workshop were to provide a comparison between non-Apis bees and honey bees, and to use the best available science to summarize how routes of exposure may vary across different bee groups.

Furthermore, participants identified potential non-Apis species that could be used in future risk assessments, as they may experience routes of exposure not considered by honey bee assessments. Participants generally agreed that non-Apis model species for pesticide assessment would necessarily be limited to those whose biology is well-known and that are commercially available with established husbandry practices. Participants further agreed on the importance of identifying bee species that are generalist or polylectic foragers, because specialization on single plant families or crops would restrict any broader understanding of exposure. This decision narrowed our scope to cavity-nesting Megachilids – Osmia spp., and the alfalfa leafcutting bee, Megachile rotundata, as representative solitary bees, and to bumble bees (Bombus impatiens, B. ignitus and B. terrestris) as social non-Apis bee surrogates. Stingless bees were also discussed and considered, although currently, relatively little is known regarding their husbandry and commercial management for pollination services (see Slaa et al., 2006). Further studies of stingless bees and how to incorporate them into pesticide assessments were identified as an important research gap to be addressed in the future. Participants acknowledged that stingless bee species display a wide diversity of behaviors that can potentially have dramatic impacts on exposure risk in different settings: This may complicate efforts to assess exposure for all stingless bees if using only one particular stingless bee surrogate.

Consideration was also given to the solitary, ground-nesting alkali bee, N. melanderi, which is managed to pollinate alfalfa grown for seed in the northwestern United States. Although husbandry practices in a laboratory setting have not been perfected, this native bee is the only intensively managed ground-nesting species that could potentially be considered for pesticide exposure assessment screening. The use of this species would require further evaluation to identify methods that could support future risk assessment needs. However, the alkali bee was incorporated into this document because of its value as a managed bee that possesses a different nesting behavior from other species previously discussed, largely due to its intimate relationship with agricultural soil. The majority of wild bee species are ground-nesting, or spend at least part of their life cycle in underground cavities (e.g., most bumble bee queens), so the consideration of pesticide exposure from soil resulting from bee nesting and hibernation is important.

Workshop participants discussed that a ‘global harmonization’ in current risk assessment procedures would avoid redundancies in evaluations and reduce overall registration costs as they relate to exposure assessment. The workshop ended with three ‘breakout’ group meetings, where basic conceptual models and preliminary exposure questions were discussed to quantify exposure via the soil/mud and plant surface routes explored below. These conceptual models were intended to serve as a basis of comparison with honey bee exposure estimates, and are shared and discussed further in the accompanying three bee review publications that also emerged from the workshop.

Exposure risk scoring

A key outcome of the workshop was the comparison of the relative importance of different potential exposure routes among the proposed bee taxa. Table 1 summarizes separate scores for honey bees, bumble bees, mason bees (Osmia spp.), the alfalfa leafcutting bee (M. rotundata), the alkali bee, and stingless bees ranging from zero (low or no known risk) to four (high risk) for each potential route of exposure identified. These exposure routes include particles in air (dust and spray), nectar, pollen, mud/soil, honeydew, wax, water, guttation fluid, plant surfaces and propolis/resin. The scoring system was inspired by the risk characterization scheme presented in the European Food Safety Authority scientific opinion (EFSA 2012). In addition to presenting a relative value for the likelihood of exposure, the type of exposure is specified (i.e., oral versus contact), and separate values are presented for likelihood of exposure during adult and larval stages. The resultant table considers the relative likelihood of exposure across bee species (rather than within species) for each route of exposure considered. Comparisons of relative likelihood for exposure within each bee subgroup are presented in accompanying reports from this workshop.

Table 1.

The likelihood of pesticide exposure, by route, to commercially important Apis and non-Apis pollinators to be considered for future risk assessment protocols.

Exposure Route Life Stage A. mellifera Bombus spp. Osmia spp. M. rotundata N. melanderi Meliponini
Air Particles Adults 4/0/1 4/4 4 4 4 4
(Contact) Larvaea 0 0 0 0 0 0

Nectar Adults 4/3/2 4/4 4 4 4 4
(Oral) Larvaeb 4 4 4 4 4 3

Pollen Adults 1/3/1 4/4 4 4 4 4
(Oral) Larvae 4 4 4 4 4 4

Mud/Soil Adultsc 0/0/0 1/2 2 0 4 4
(Contact) Larvae 0 0 1 0 4 4

Wax d Adults 1/3/3 1/1 0 0 0 4
(Contact) Larvae 4 4 0 0 0 4

Water Adults 4/1/1 1/1 1 1 1 4
(Oral) Larvae 1 0 0 0 2 2

Plant Surfaces Adults 3/0/0 3/3 3 4e 3 4
(Contact) Larvae 0 0 0 4f 0 0

Propolis/Resin Adults 3/1/1 1/1 0 0 0 4
(Contact) Larvae 0 0 0 0 0 4

Honeydew g Adults 4/2/0 1/1 0 0 0 4
(Oral) - - - - - -

Gutattion Fluid h Adults 1/1/1 0/0 0 0 0 1
(Oral) - - - - - -

Designated values rank from 0 (marginal or no likelihood of exposure) to 4 (high likelihood of exposure) for both adults and larval bees. Under each exposure route identified, the primary category of exposure (contact or oral) is specified. For Apis mellifera, relative values are provided for foragers, in-hive bees and overwintering bees, respectively. For Bombus spp., values are provided for foraging workers and queen bees, respectively. Values presented here are intended for cross-subgroup comparison (rows), and not within-subgroup comparisons (columns).

a,b

All larvae would also experience contact exposure through both nectar and pollen

c

Adult Osmia spp., N. melanderi, and Meliponini would also experience oral exposure through mud/soil

d

Exposure to wax would occur only for pollinators that produce wax (Apis mellifera, Bombus spp., and Meliponini)

e

M. rotundata adults are subject to both oral and contact exposure via plant surfaces, as they forage upon leaf pieces with which to line their nests

f

Larval M. rotundata exposure via plant surfaces is primarily by contact and not oral exposure, as they reside in leaf-lined cells for the entirety of their development

g,h

Larval exposure to pesticides in honeydew and guttation fluid were presumed to be minimal and were not discussed at length by workshop participants

For the purposes of this workshop, no consideration of prepupal or pupal stages were made. Although prepupae and pupae do not feed, they may still experience chronic contact exposure with pesticide residues in the environment around them via wax, leaf pieces (M. rotundata), mud partitions (Osmia spp.), or soil (N. melanderi). Workshop participants agreed that potential movement of pesticide residues across the pupal case, as opposed to the larval cuticle, would require investigation.

Additionally, bees not only experience differential exposure resulting from caste or developmental stage, but seasonality and climatic conditions can also influence exposure. As is the case with honey bees, bumble bees and stingless bees express seasonal- and climate-induced changes in behavior that can alter the frequency and/or duration of exposure to pesticide residues in the environment. The timing of solitary bee development also depends on environmental conditions and on the life history of each species. As a result, the relative likelihood of exposure can change dramatically depending on the time of year a pesticide is applied, how it is applied, the matrix it interacts with, and its persistence over time in the environment. Such differences complicate the interpretation of the different routes of exposure but are nonetheless integral to decisions made in the development of future assessment protocols.

Exposure Routes

Particles in air

‘Particles in air’ refer to both bee inhalation of and cuticular exposure to volatilized particles in the air that result from either abrasion of treated seed coatings during planting or pesticide spray drift. Exposure of adult bees via particles in air will primarily occur as contact exposure, and efforts to quantify exposure relative to others discussed in this document (e.g., ingestion of nectar and pollen) have been limited (Tapparo et al., 2012, Girolami et al., 2012).

For foraging adults, the likelihood of contact exposure for all bee groups ranked 4 (Table 1). During field application, it would be difficult for any adult bee, Apis and non-Apis species alike, to avoid exposure to pesticides in the air while actively foraging. Risk assessments currently in place for honey bees take this exposure route into account, though this has not been widely considered or discussed for non-Apis bees. Because bee larvae are sheltered from the environment, they were considered to experience no direct exposure via this route. Larvae may be exposed to pesticide spray drift, indirectly, as it descends upon soil and adjacent foliage that may ultimately comprise the nesting materials surrounding immature bees. Similarly, any indirect adult oral exposure to particles in air would largely be a consequence of where those particles land in the environment. Therefore, these considerations are incorporated into the forthcoming sections (e.g., see ‘mud’ and ‘plant surfaces’).

Because adult bees from all species are considered to have a high likelihood of exposure to particles in air, current considerations in place for honey bees likely serve as sufficient surrogates for other non-Apis bee species. Based on existing literature, there was general consensus that non-Apis larval exposure to particles in air would not experience exposure exceeding that for honey bee larvae. Therefore, such exposure to non-Apis bees may not warrant further consideration in future risk assessments.

Diet (nectar and pollen)

Dietary exposure through consumption of pollen and nectar was an important exposure route discussed, and received the highest risk exposure score across all bee species. The diet of both larval and adult honey bees changes dramatically with age, particularly the ratio and rate of consumption of nectar and pollen (Pirk et al., 2010; Paoli et al., 2014). It is likely that similar changes also occur in other non-Apis bee species, as is seen during the maturation of ovaries in adult female bees (Stabler et al., 2015, Cane 2016). For honey bees, estimates of daily diet composition and consumption for larvae and adults, while conservative, have been generally agreed upon in the literature (USEPA 2014; Sanchez-Bayo and Goka, 2014). For solitary and stingless bees, only limited information is available on average provision composition and consumption rates, which limits the ability to quantify dietary exposure (such data are more readily available for bumblebees; see Cresswell et al., 2012 and Laycock et al., 2012). This represents a substantial and important data gap limiting our assessment of the extent to which current risk assessments are representative of non-Apis bee species. This information may be obtained by feeding and tracking the movement of radiolabeled glucose in bee species (Nixon and Ribbands, 1952), though results will likely vary according to bee body size and energy needs. The challenge for larvae is their continuous growth and changing food consumption rates with time. A possible solution to this concern could be to determine exposure as a concentration in food, and consider whether total consumption per unit body weight in non-Apis bees are likely to be substantially different than those for honey bees (Botías et al. 2017).

Honey bees, bumble bees and stingless bees are progressively fed throughout the larval stage, while solitary bees consume a single mass provision, consisting of pollen and nectar, which is constructed by the mother bee prior to egg-laying. This can change the nature of exposure in the larval diet, because for any single larvae, mass provisioning usually reflects a pollen and nectar collected over a short period of time (ca. one day for some species; Hackwell, 1967; Cane and Pitts-Singer, 2011). Additionally, consumption of a mass provision involves an extended duration of larval contact with the provision for solitary bees, because larvae are essentially on top of or leaning against the pollen mass throughout its consumption.

The ratio of nectar and pollen in solitary bee provisions is variable, both within and among species, and can be challenging to manipulate in a laboratory setting. Therefore, it would be important to quantify the pollen-to-nectar ratio of provisions when developing protocols for bees. Quantified ratios can contribute to calculations of a compound in question in pollen and nectar, for comparison with a concentration-based toxicity endpoint. For example, alfalfa leafcutting bee provisions are generally ‘wetter’ (containing more nectar) than blue orchard bee provisions, although the consistency of provisions can vary even within a species depending upon available floral resources and climate. This variability may influence the relative likelihood of exposure to pesticides present in nectar and pollen (Kopit and Pitts-Singer, 2018). For solitary bee larval exposure assessments, it would also be difficult to consider nectar and pollen exposure risk separately, since they are effectively impossible to tease apart within a provision. Further, the pollen-to-nectar ratio in solitary bee provisions is largely uncharacterized for both managed and wild species. For these reasons, nectar and pollen are both ranked as 4 (high likelihood of exposure) in the evaluation (Table 1).

Little is known about the dietary and energetic needs of bumble bee queens, which limits our ability to assess their exposure from ingestion of pollen and nectar. Bumble bee queens are expected to be particularly vulnerable to exposure prior to, and during, nest founding, when the queen is gathering her own nourishment (Baron et al., 2017a,b). Her nectar or pollen consumption rates after colony establishment, when other workers are feeding her, also are largely unknown.

As is the case for honey bees, stingless bee larvae are fed in part by glandular secretions from adults; although, little is known about the composition or nutritional value of these secretions. It is possible that this feeding strategy renders stingless bee larvae less susceptible to exposure to pesticides from the environment due to post-ingestion processing by the workers that feed them. Knowledge about even the biology and diet of stingless bee larvae is largely unknown, so attributing values to indicate the likelihood of exposure in a larval context is difficult for nectar and pollen.

Nectar

Floral preferences will have obvious consequences for bee exposure. Honey bees tend to select nectar sources that provide the greatest sugar reward. This may not always be the case with solitary bees and requires further investigation. The greatest consumption rate of nectar-foraging worker honey bees is based on the equation proposed by Rortais et al. (2005):

Dnectar=TSFDFP
  • T = the average number of foraging trips made per day

  • SF = the quantity of sugar required for flight

  • D = the duration of each foraging trip

  • F = the fraction of the foraging trip spent flying

  • P = the percentage of sugar in nectar

This equation accounts for five different variables that influence the calculation of the adult consumption rate for nectar (approximate values obtained for honey bees by Rortais et al. (2005) are in parentheses): the average number of foraging trips per day (10), the amount of sugar required for flight (8 – 12 mg/hr.), the duration of each foraging trip (0.5- 1.33 hr.), the proportion of the foraging trip spent flying (0.8), and finally, the amount of sugar present in nectar (a generally accepted average of 30%).

It is important to know the typical number of foraging trips completed in a day by non-Apis species. Ten foraging trips per day is considered typical for honey bees. However, this may not apply to other bee species, especially among those that lack task allocation exhibited by social colonies. For example, female solitary bees typically spend their adult lifespan provisioning nest cells as individuals (Gathmann and Tscharntke, 2002).

In honey bees, the crop (or honey stomach) typically contains nectar that is about 30% sugar. Abrol (2005) estimated that honey bees consume 37.35 mg sugar per hour, which may be appropriate and reliable as a metric for quantifying exposure until more information related to sugar intake is obtained. However, our limited knowledge of sugar consumption by adult honey bees continues to leave uncertainty whether exposure estimates for honey bees would be sufficiently protective for non-Apis bees. To address this uncertainty, additional information is needed on nectar preferences and feeding durations for honey bees and other bee species (Willmer and Stone, 2004; Kim et al., 2011). Current estimates of bumble bee colony consumption rates includes all sugar consumed over the course of a colony lifespan, from founding stages to colony end (Rotheray et al., 2017). Such estimates will vary considerably, however, depending upon bee species, bee size, colony size (where applicable), nesting season and other factors related to species range and the availability of nutritional resources. Additionally, measures of a colony’s metabolic needs in a laboratory are typically lower than for a wild colony (as is upheld in many biological systems) because confined colonies do not have to forage great distances and environmental conditions are less variable.

Solitary bee larval exposure (resulting from combined oral ingestion and physical contact) from nectar likely exceeds that for social bees, because for social species nectar is processed by adult workers before reaching immature bees (Nixon and Ribbands, 1952). The enzymatic breakdown associated with processing of sugars and possibly pesticides in nectar performed by workers in social colonies may mitigate some exposure for larvae. Consumption of ‘pre-processed’ sugars also buffers exposure for younger nurse bees. Nonetheless, this uncertainty is covered in the current honey bee exposure assessment, as the conservative screening level (Tier 1) exposure estimates consider only unprocessed nectar for both larvae and adults. Mechanisms of nectar and sugar processing in stingless bees are largely unknown. Thus, there was some reluctance to categorize the likelihood of exposure for stingless bees by caste at this time because of the current lack of available information. Until these aforementioned nectar-related research gaps can be filled, there was general consensus among participants that honey bees represent a reasonably appropriate surrogate for other bee species’ likelihood of exposure to chemicals in nectar.

Pollen

Nearly all bee larvae require a diet relatively rich in pollen, consuming substantially more than adults. Honey bee pollen is processed by adults (nurse bees) as it is collected from floral resources that older adult foragers bring back to the hive. The hypopharyngeal gland secretions of honey bees may limit the extent to which pesticide residues are passed on to larvae. Nonetheless, the current protocol for screening level (Tier 1) exposure assessment for honey bees is conservative and evaluates only unprocessed pollen for both larvae and adults. Compared to other adult honey bee workers, nurse bees eat relatively more pollen, which may increase their likelihood of exposure for ingestion of pesticides and/or associated metabolites than foragers (Rortais et al. 2005, Sanchez-Bayo and Goka, 2014). This scenario may hold true for stingless bees, as they also exhibit similar age polyethism. However, as task allocation is less fixed among workers in a bumble bee colony, one could predict they have more similar likelihoods of exposure, although larger workers are typically foragers and smaller workers stay inside the nest (Spaethe and Weidenmüller, 2002; Goulson et al., 2002). Because pollen examined in current Tier 1 assessments is unprocessed, any reductions in concentration due to processing observed for honey bees are not considered in those exposure estimates. All bee groups evaluated possess a high likelihood of adult and larval oral and contact exposure, and honey bees may serve as reasonably effective candidates in evaluating the likelihood of exposure to chemicals in pollen until more information is obtained regarding the quantity of pollen consumed by non-Apis species.

Mud/Soil

Presence of pesticides at the soil surface or in the soil matrix is likely to affect species of social and solitary bees differently due to differences in nesting behavior. Because honey bees do not use mud or soil in nest construction, nor reside in underground tunnels or cavities, the consideration of non-Apis bee interactions with soil is underrepresented in current pesticide risk assessment protocols. This exposure route is also highly dependent upon the chemical properties of pesticides. Pesticides can infiltrate soils in many ways, such as in runoff from agricultural fields, as part of a seed treatment, direct application as soil drenches, foliar applications, etc. Tilling versus no-till practices may also influence the likelihood of exposure (Elliott et al. 2000). Exposure to pesticides in soil is influenced by how the pesticide is applied, how the pesticide interacts with the soil matrix and climatic conditions (Kopit and Pitts-Singer, 2018).For ground-nesting bees, species exhibit different nesting preferences for soil type, which may lead to even greater variation in likelihood of exposure.

Soil type will result in differences in uptake for various chemicals. The extent to which a pesticide interacts with a matrix (e.g., soil, water) depends on the chemical and physical characteristics of the chemical and associated matrix. Many extensive and complex models have been developed to predict the partitioning of chemicals in different soil matrices (reviewed in Gevao et al., 2000; Schaeffer et al., 2010). However, determining the extent to which chemicals partition in soil over time in agricultural environments remains a challenge, as agricultural practices (e.g., irrigation, tilling) and environmental conditions (e.g., precipitation, runoff) can influence the movement of a chemical through the soil profile. One option is to consider a worst-case scenario in which a ‘reasonably conservative’ estimate for exposure in a fully saturated soil is calculated; however, even then, exposure will vary significantly according to soil moisture level. In addition, the life cycle of developing larvae and pupae underground can undergo variable developmental trajectories that occupy different ranges or periods of time. Over time, there would be a gradual degradation/dissipation of materials in the soil matrix; such degradation may be difficult to predict in a reasonably conservative way.

Both adult and larval stages of ground-nesting solitary bees and bumble bees have the highest likelihood of chemical exposure in soil. Adult females must dig and construct brood cells in soil, resulting in repeated contact exposure, while larvae hatch and develop underground, resulting in chronic, long-term exposure. Some Osmia spp. also rely upon mud to build partitions in their nests, although this genus does not generally reside underground (Cane et al., 2007). Stingless bees have also been known to collect mud with to build their nests. Larvae of ground-nesting bees may experience chronic contact with soil. However, this contact would likely be moderated by brood cell linings that female ground nesting bees often apply to the inner chamber of their nest(s), using glandular wax or other secretions. The composition, scope and functional role of these inner linings are other important research gaps to be addressed.

For bumble bees, overwintering queens are at a potentially high risk of exposure to materials in soil. Queens typically overwinter underground, residing in the soil for several months prior to colony initiation. The importance of bumble bee queens to the colony and their extended lifespan makes research in this area of critical importance. However, as with other ground-dwelling species (social and solitary), measuring the transfer rate of materials across soil and into an overwintering adult or prepupa will depend on the chemical, the soil type, type of chemical application, moisture level in the soil, and temperature.

There was general agreement among workshop participants that honey bee risk assessment procedures fall short in quantifying risk for non-Apis bee species reliant on soil, or mud, for both larval and adult bees. The intimate association of both larval and adult N. melanderi and ground-nesting stingless bees with soil puts them at the highest likelihood (ranked ‘4’) of contact exposure. For O. lignaria and N. melanderi, adult oral manipulation and/or collection of soil is imperative for reproduction and could lead to a low, but not insignificant, degree of oral exposure to pesticides within soil (ranked ‘1’). This exposure route was highlighted as worthy of consideration in future risk assessment protocols.

Honeydew

There is not much evidence in the literature for bees foraging on honeydew secretions from other insects. Honey bees have been occasionally known to forage on honeydew, and there is some knowledge of bumble bees and stingless bees foraging on honeydew (Schuster, 1981; Bishop, 1994; Koch et al., 2011). However, to our knowledge, this behavior has not been observed for other bee species. Workshop participants concluded that any bee foraging upon honeydew would be doing so out of necessity, and not preference (for example, if available floral resources were extremely limited). Thus, the likelihood of exposure for bees from foraging on contaminated honeydew would be minimal compared to the levels of exposure via in other routes. Further, any exposure to pesticides via honeydew would be limited to secretions in which chemicals were not already metabolized by the honeydew-producing insect. It was concluded that current knowledge provides little to no evidence to suggest that honeydew is an appreciable route of exposure for Apis or non-Apis bees.

Wax

Honey bees, bumble bees and stingless bees all reside, store food and rear offspring in wax. Thus, both larvae and adults are susceptible to contact exposure to pesticide residues embedded in wax. Although solitary bees may line their nests with hydrophobic, wax-like secretions, these secretions are more likely to provide a protective barrier against pesticide exposure, rather than acting as a source of contaminants. However, the function of glandular secretions in preventing or minimizing exposure to environmental contaminants has not been studied specifically. Because honey bee, bumble bee and stingless bee larvae develop in individually enclosed wax cells, their potential for contact with chemicals in wax ranked high (3). Adult honey bees, bumble bees, and stingless bees also spend much of their lives residing on potentially contaminated wax (as noted in table 1). However, because adult bumble bee workers do not overwinter as honey bees and stingless bees do, their relative likelihood of exposure is lower. Oral manipulation of wax was determined not to contribute substantially to pesticide exposure. Based on current knowledge, workshop participants concluded that honey bee risk assessments for chemical exposure in wax are reasonably conservative for non-Apis bees more broadly. Notably, while pesticide residue accumulation in bumble bee and stingless bee wax has been acknowledged in the literature, it has not been as thoroughly characterized for these species as it has been for honey bees (Cabrera et al., 2015), and would benefit from further research.

Water

One challenge in evaluating the likelihood of exposure to pesticides in honey bees is quantifying typical water consumption rates. Current methods to evaluate exposure are scarce, and it is reasonable to assume that water intake will vary by bee species, season, climate, temperature, etc. (Kühnholz and Seeley, 1997). Exposure through water would likely be lower than that observed for other components of any bee diet, i.e., pollen and nectar (Sanchez-Bayo and Goka, 2014).

There was a consensus that although many solitary bees and bumble bees collect and imbibe water, it is infrequent when compared to honey bees that require water to cool the hive and manipulate honey. Little is known about water collection behavior and frequency in stingless bee colonies, which makes characterizing exposure risk for this group difficult. Until further studies are conducted to quantify water foraging in stingless bees, workshop participants tentatively supported the use of honey bees as an appropriate surrogate for non-Apis bee species’ likelihood of exposure via water.

There was considerable workshop discussion about whether larvae of ground-nesting species come into direct contact with water other than moisture that turns soil to mud. There may be a distinction between water that moves through a soil profile and ‘mud’, which would have consequential impacts on the fate of particular chemicals in question. A potential overlap in some of these exposure routes is possible and may vary depending upon the hydrophilicity of the chemical compound itself and qualities of the soil matrix. For hydrophobic compounds, sorption to substrates will reduce the availability via contact exposure, but the relative importance of oral exposure versus contact exposure may be increased.

Guttation fluid

Honey bees appear to collect guttation fluid rarely, and very little to no collection is expected for non-Apis bees, although this has not been widely reported on in the literature (Reetz et al., 2016). This route of exposure would only be relevant to the use of systemic pesticides, such as neonicotinoids, and to pollinator-attractive crops on which such chemicals are used (e.g., maize and oilseed rape/canola). Additionally, the extent to which guttation occurs depends on the crop itself and environmental conditions. Guttation fluid is considered a minor exposure route as compared to dietary exposure through pollen and nectar consumption. There was participant consensus that this route probably does not pose a large exposure source for bees.

Plant surfaces

‘Plant surfaces’ is a term that includes chemically-exposed leaves, stems, and blossoms. Bees encounter these matrices while foraging for various resources. Contact with the blossom occurs as a consequence of collecting nectar and pollen, along with indirect interactions with stems, leaves and other foliage. In most cases, workshop participants concluded that current estimates for exposure to adult honey bees can serve as an appropriate surrogate for other, non-Apis species. However, some bees, including M. rotundata, use their mandibles to cut and carry leaf pieces to line their nest cells. This constitutes both adult contact and oral exposure. Larvae that are developing inside nests lined with leaf pieces are also subject to exposure to materials on the foliage. Thus, leafcutting bee larvae were considered to have a comparatively high likelihood of contact exposure to pesticides via plant surfaces. In addition, the duration of larval and pupal development inside the nest will influence exposure; individuals with a shorter larval period may be subject to less exposure than individuals taking longer to reach the pupal stage. Finally, many non-Apis bees may experience exposure to unique plant materials (e.g., floral oils or resins) that are not widely used by other groups of bees.

Propolis/Resin

As with honey bees, stingless bees collect plant resins and incorporate them into their nests. Although many species of solitary bees have also been observed collecting resin, the solitary bee species discussed during this workshop do not; thus, they express a relatively lower likelihood for exposure from this source. Both adults and larvae of resin-foraging species are subject to exposure to pesticides therein.

Additional research needs

Two additional research gaps, identified below, frequently came up over the course of the workshop..

Allometric scaling to predict toxicological response

An important and recurring question throughout the workshop related to whether or how bee toxicological responses scale to body size. Intuitively, one can predict that smaller bees would be more susceptible to the effects of exposure than larger bees, and there is some evidence to support this theory (Devillers et al., 2003, Arena and Sgolastra, 2014, Thompson, 2016). Cresswell et al. (2012), however, reports a confounding trend; such variability may be a consequence of differential sensitivity across species, and the specific mode of action of the pesticide in question (Heard et al. 2017), or potentially the quality or thickness of the insect cuticle. In considering allometric scaling, what would be an appropriate, discrete metric for body size? Weight, body surface area, intertegular span or some other morphometric measurement may be appropriate, although little information of this kind is typically reported in the literature (Cane 1987). Although this discussion of allometric scaling moved beyond the scope of the workshop (addressing more ‘hazard’ than ‘exposure’), participants recognized that these considerations could also be helpful in extrapolating pesticide toxicity from honey bees to non-Apis bee species, particularly if there is a reliable toxicity metric for extrapolation according to body parameters (e.g., how mass-specific metabolic rates increase with decreasing body size; Chown et al., 2007). Use of body weight alone is unreliable because of the inherent range of average bee weights within and among species, which is largely undocumented.

Would body surface area be an appropriate metric for adjusting exposure predictions? Although this measure may present an avenue for reliable extrapolation of data from honey bee tests to other bees, in most cases of pesticide exposure, bee surface contamination is not uniform across the entire bee body. For example, when considering direct overspray as the route of exposure to adult bees, it can be assumed that contact will be limited to the dorsal surfaces of their body (including wings; Poquet et al. 2015), making dose difficult to quantify in this context. Some efforts to characterize body surface area of bee species are already underway (Kühsel et al. 2016).

Metabolic needs required for flight

The metabolic needs of bees to power flight are also relatively understudied. Clearly, nectar is required for foraging adult bees to power flight (among other activities), and consumption rates will change with respect to flight demands of a colony in the case of honey bees, stingless bees and bumble bees. Furthermore, metabolic calculations in social colonies may be unique to various castes and the tasks performed by workers. Because female solitary bees are self-sufficient and exercise the same behaviors daily, there is likely less variation to be expected in their metabolic needs, regardless of age. Existing literature demonstrates that smaller bee foraging ranges do not always scale consistently with larger bees (Greenleaf et al. 2005). Therefore, energy consumption will likely be difficult to estimate in relation to body size. Furthermore, there is some variation across taxa regarding thermoregulation. For example, honey bees and bumble bees thermoregulate their nests, while solitary bees do not (Morato and Martins, 2006). Finally, understanding consumption rates across non-Apis bee species is relevant to the reproductive success of egg-laying individuals (queens in social colonies, and all female solitary bees). Thus, bioenergetics across candidate surrogate species will be a valuable area to direct future research efforts.

Footnotes

Publisher's Disclaimer: Disclaimer: The research described in this article has been reviewed by the National Health and Environmental Effects Research Laboratory, U.S. Environmental Protection Agency, and approved for publication. Approval does not signify that the contents necessarily reflect the views and policies of the Agency, nor does the mention of trade names of commercial products constitute endorsement or recommendation for use.

References

  1. Abrol DP 2005. Pollination energetics. J. Asia-Pac. Entomol 8: 3–14. [Google Scholar]
  2. (APVMA) Australia Pesticides and Veterinary Medicines Authority. 2015. Roadmap for Insect Pollinator Risk Assessment in Australia. http://apvma.gov.au/node/18536 (last accessed 09/27/2016).
  3. Arena M, and Sgolastra F. 2014. A meta-analysis comparing the sensitivity of bees to pesticides. Ecotoxicology 23: 324–334. [DOI] [PubMed] [Google Scholar]
  4. Baron GL, Jansen VAA, Brown MJF and Raine NE. 2017a. Pesticide reduces bumble bee colony initiation and increases probability of population extinction. Nature Ecol. Evol 1: 1308–1316. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Baron GL, Raine NE and Brown MJF. 2017b. General and species-specific impacts of a neonicotinoid insecticide on the ovary development and feeding of wild bumble bee queens. Proc. Roy. Soc. B – Biol. Sci 284: 20170123. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Bishop JA 1994. Bumble bees (Bombus hypnorum) collect aphid honeydew on stone pine (Pinus pumila) in the Russian far east. J. Kans. Entomol. Soc 67: 220–222. [Google Scholar]
  7. Botías C, David A, Hill EM, and Goulson D. 2017. Quantifying exposure of wild bumblebees to mixtures of agrochemicals in agricultural and urban landscapes. Environ. Pollut 222: 73–82. [DOI] [PubMed] [Google Scholar]
  8. Cabrera AR, Almanza MT, Cutler GC, Fischer DL, Hinarejos S, Lewis G, Nigro D, Olmstead A, Overmyer J, Potter DA, Raine NE, Stanley-Stahr C, Thompson H, and van der Steen J. 2015. Initial recommendations for higher-tier risk assessment protocols for bumble bees, Bombus spp. (Hymenoptera: Apidae). Integ. Environ. Asses 12: 222–229. [DOI] [PubMed] [Google Scholar]
  9. Cane JH 1987. Estimation of bee size using intertegular span (Apoidea). J. Kans. Entomol. Soc 60: 145–147. [Google Scholar]
  10. Cane JH, Griswald T, and Parker FD. 2007. Substrates and materials used for nesting by North American Osmia Bees (Hymenoptera: Apiformes: Megachilidae). Ann. Entomol. Soc. Am 100: 350–358. [Google Scholar]
  11. Cane JH 2016. Adult pollen diet essential for egg maturation by a solitary Osmia bee. J. Insect Physiol 95: 105–109. [DOI] [PubMed] [Google Scholar]
  12. Cane JH, and Pitts-Singer TL. 2011. The alfalfa leafcutting bee, Megachile rotundata: The world’s most intensively managed solitary bee. Annu. Rev. Entomol 56: 221–237. [DOI] [PubMed] [Google Scholar]
  13. Cham K. de O., Rebelo RM, Oliveira R. de P., Ferro AA, Vianasilva F. E. de C., Borges L. de O., Saretto COSD, Tonelli CAM, and Macedo TC. 2017. Manual de avaliação de risco ambiental de agrotóxicos para abelhas. Brasília: Ibama/Diqua. http://www.ibama.gov.br/phocadownload/agrotoxicos/reavaliacao-ambiental/2017/2017-07-25-Manual-IBAMA-ARA-Abelhas-IN0217-WEB.pdf [Google Scholar]
  14. Chown SL, Marais E, Terblanche JS, Klok CJ, Lighton JRB, and Blackburn TM. 2007. Scaling of insect metabolic rate is inconsistent with the nutrient supply network model. Funct. Ecol 21: 282–290. [Google Scholar]
  15. Cresswell JE, Page CJ, Uygun MB, Holmbergh M, Li Y, Wheeler JG, Laycock I, Pook CJ, Hempel de Ibarra N, Smirnoff N, and Tyler CR. 2012. Differential sensitivity of honey bees and bumble bees to a dietary insecticide (imidacloprid). Zoology 115: 365–371. [DOI] [PubMed] [Google Scholar]
  16. Devillers J, Decourtye A, Budzinski H, Pham-Delègue MH, Cluzeau S, and Maurin G. 2003. Comparative toxicity and hazards of pesticides to Apis and non-Apis bees. A chemometrical study. SAR QSAR Environ. Res 14: 389–403. [DOI] [PubMed] [Google Scholar]
  17. (EFSA) European Food Safety Authority. 2012. Panel on plant protection products and their residues (PPR); scientific opinion on the science behind the development of a risk assessment of plant protection products on bees (Apis mellifera, Bombus spp. and solitary bees). EFSA J. 10: 2668. [Google Scholar]
  18. (EFSA) European Food Safety Authority. 2013. Guidance on the risk assessment of plant protection products on bees (Apis mellifera, Bombus spp. and solitary bees). EFSA J. 11: 3295. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Elliott JA, Cessna AJ, Nicholaichuk W, and Tollefson LC. 2000. Leaching rates and preferential flow of selected herbicides through tilled and untilled soil. J. Environ. Qual 29: 1650–1656. [Google Scholar]
  20. Garibaldi LA, Steffan-Dewenter I, Winfree R, Aizen MA, Bommarco R, Cunningham SA, Kremen C, Carvalheiro LG, Harder LD, Afik O, Bartomeus I, Benjamin F, Boreux V, Cariveau D, Chacoff NP, Dudenhoffer JH, Freitas BM, Ghazoul J, Greenleaf S, Hipolito J, Holzschuh A, Howlett B, Isaacs R, Javorek SK, Kennedy CM, Krewenka K, Krishman S, Mandelik Y, Mayfield MM, Motzke I, Munyuli T, Nault BA, Otieno M, Peterson J, Pisanty J, Potts SG, Rader R, Ricketts TH, Rundlof M, Seymour CL, Schuepp C, Szentgyörgyi H, Taki H, Tscharntke T, Vergara CH, Viana BF, Wanger TC, Westphal C, Williams NM, and Klein AM. 2013. Wild pollinators enhance fruit set of crops regardless of honey bee abundance. Science 339: 1608–1611. [DOI] [PubMed] [Google Scholar]
  21. Gathmann A, and Tscharntke T 2002. Foraging ranges of solitary bees. J. Anim. Ecol 71: 757–764. [Google Scholar]
  22. Gevao B, Semple KT, and Jones KC. 2000. Bound pesticide residues in soils: a review. Environ Pollut. 108: 3–14. [DOI] [PubMed] [Google Scholar]
  23. Girolami V, Marzaro M, Vivian L, Mazzon L, Giorio C, Marton D, and Tapparo A. 2012. Aerial powdering of bees inside mobile cages and the extent of neonicotinoid cloud surrounding corn drillers. J. Appl. Entomol 137: 35–44. [Google Scholar]
  24. Goulson D, Peat J, Stout JC, Tucker J, Darvill B, Derwent LC, and Hughes WOH. 2002. Can alloethism in workers of the bumble bee, Bombus terrestris, be explained in terms of foraging efficiency? Anim. Behav 64: 123–130. [Google Scholar]
  25. Greenleaf SS, Williams NM, Winfree R, and Kremen C. 2007. Bee foraging ranges and their relationship to body size. Oecologia 153: 589–596. [DOI] [PubMed] [Google Scholar]
  26. Hackwell GA 1967. The biology and behavior of the alkali bee Nomia melanderi Cockerell (Hymnoptera: Apoidea). Ph.D. dissertation, University of Oregon, Corvallis, OR. [Google Scholar]
  27. Heard MS, Baas J, Dorne J-L, Lahive E, Robinson AG, Rortais A, Spurgeon DJ, Svendsen C, and Hesketh H. 2016. Comparative toxicity of pesticides and environmental contaminants in bees: Are honey bees a useful proxy for wild bee species? Sci. Total Environ 578: 357–365. [DOI] [PubMed] [Google Scholar]
  28. Kim W, Gilet T, and Bush JWM. 2011. Optimal concentrations in nectar feeding. Proc. Nat. Acad. Sci 108: 16618–16621. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Koch H, Corcoran C, and Jonker M. 2011. Honeydew collecting in Malagasy stingless bees (Hymenoptera: Apidae) and observations on competition with invasive ants. Afr. Entomol 19: 31–41. [Google Scholar]
  30. Kopit AM, and Pitts-Singer TL. 2018. Routes of pesticide exposure in solitary, cavity-nesting bees. Environ. Entomol 47: 499–512. [Google Scholar]
  31. Kühnholz S, and Seeley TD. 1997. The control of water collection in honey bee colonies. Behav. Ecol. Sociobio 41: 407–422. [Google Scholar]
  32. Kühsel S, Brückner A, Schmelzle S, Heethoff M, and Blüthgen N. 2016. Surface area–volume ratios in insects. Insect Sci. 24: 829–841. [DOI] [PubMed] [Google Scholar]
  33. Laycock I, Lethall KM, Barratt AT, and Cresswell JE. 2012. Effects of imidacloprid, a neonicotinoid pesticide, on reproduction in worker bumble bees (Bombus terrestris). Ecotoxicology 21: 1937–1945. [DOI] [PubMed] [Google Scholar]
  34. Morato EF, and Martins RP. 2006. An overview of proximate factors affecting the nesting behavior of solitary wasps and bees (Hymenoptera: Aculeata) in preexisting cavities in wood. Neotrop. Entomol 35: 285–298. [DOI] [PubMed] [Google Scholar]
  35. Nixon HL, and Ribbands CR. 1952. Food transmission within the honey bee community. Proc. Royal Soc. B: Biol. Sci 140: 43–50. [DOI] [PubMed] [Google Scholar]
  36. Paoli PP, Donley D, Stabler D, Saseendranath A, Nicolson SW, Simpson SJ, and Wright GA. 2014. Nutritional balance of essential amino acids and carbohydrates of the adult worker honeybee depends on age. Amino Acids 46: 1449–1458. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Peterson SS, and Artz DR. 2014. Production of solitary bees for pollination in the United States. pp. 653–681, In Morales-Ramos JA, Guadalupe Rojas M, Shapiro-Ilan DI, Mass production of beneficial organisms: Invertebrates and entomopathogens. Academic Press, London, United Kingdom. [Google Scholar]
  38. Pirk CWW, Boodhoo C, Human H, and Nicolson SW. 2010. The importance of protein type and protein to carbohydrate ratio for survival and ovarian activation of caged honeybees (Apis mellifera scutellata). Apidologie 41: 62–72. [Google Scholar]
  39. Poquet Y, Kairo G, Tchamitchian S, Brunet J-L, and Belzunces LP. 2015. Wings as a new route of exposure to pesticides in the honey bee. Environ. Toxicol. Chem 34: 183–1988. [DOI] [PubMed] [Google Scholar]
  40. Reetz JE, Schulz W, Seitz W, Spiteller M, Zühlke S, Armbruster W, and Wallner K. 2016. Uptake of neonicotinoid insecticides by water-foraging honey bees (Hymenoptera: Apidae) through guttation fluid of winter oilseed rape. J. Econ. Entomol 109: 31–40. [DOI] [PubMed] [Google Scholar]
  41. Rortais A, Arnold G, Halm MP, and Touffet-Briens F. 2005. Modes of honeybees exposure to systemic insecticides: estimated amounts of contaminated pollen and nectar consumed by different categories of bees. Apidologie 36: 71–83. [Google Scholar]
  42. Rotheray EL, Osborne JL, and Goulson D. 2017. Quantifying the food requirements and effects of food stress on bumble bee colony development. J. Apic. Res 56: 288–299. [Google Scholar]
  43. Sanchez-Bayo F, and Goka K. 2014. Pesticide residues and bees – A risk assessment. PLOS ONE 9: e94482. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. (SAP) Scientific Advisory Panel 2012. Final FIFRA SAP Pollinator Meeting Report. SAP Minutes No. 2012–06. A set of scientific issues being considered by the Environmental Protection Agency regarding pollinator risk assessment framework. September 11 – 14, 2012, FIFRA Scientific Advisory Panel Meeting. http://www.epa.gov/sites/production/files/2015-06/documents/091112minutes.pdf [Google Scholar]
  45. Schaeffer A, Van den Brink PJ, Heimbach F, Simon P, Hoy SP, de Jong FMW, Rombkem J, Roß-Nickoll M, and Sousa JP. 2010. Semi-field methods for the environmental risk assessment of pesticides in soil. CRC Press, Boca Raton, FL: (2010) [DOI] [PubMed] [Google Scholar]
  46. Schuster JC 1981. Stingless bees attending honeydew-producing tree-hoppers in Guatemala. Fla. Entomol 64: 192. [Google Scholar]
  47. Slaa EJ, Chaves LAS, Malagodi-Braga KS, and Hofstede FE. 2006. Stingless bees in applied pollination: practice and perspectives. Apidologie 37: 293–315. [Google Scholar]
  48. Spaethe J, and Weidenmüller A. 2002. Size variation and foraging rate in bumble bees (Bombus terrestris). Insectes Sociaux 49: 142–146. [Google Scholar]
  49. Stabler D, Paoli PP, Nicolson SW, and Wright GA. 2015. Nutrient balancing of the adult worker bumble bee (Bombus terrestris) depends on its dietary source of essential amino acids. J. Exp. Biol 218: 793–802. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Tapparo A, Martin D, Giorio C, Zanella A, Soldà L, Marzaro M, Vivan L, and Giorolami V. 2012. Assessment of the environmental exposure of honeybees to particulate matter containing neonicotinoid insecticides coming from corn coated seeds. Environ Sci Technol. 46: 2592–2599. [DOI] [PubMed] [Google Scholar]
  51. Thompson HM 2016. Extrapolation of acute toxicity across bee species. Integr. Environ. Assess. Manag 12: 622–626. [DOI] [PubMed] [Google Scholar]
  52. (USEPA) U. S. Environmental Protection Agency. 2014. Guidance for assessing pesticide risks to bees. Office of Pesticide Programs, Health Canada Pest Management Regulatory Agency, California Department of Pesticide Regulation. https://www.epa.gov/pollinator-protection/pollinator-risk-assessmentguidance [Google Scholar]
  53. Willmer PG, and Stone GN. 2004. Behavioral, ecological, and physiological determinants of the activity patterns of bees. Adv. Study Behav 34: 347–466. [Google Scholar]
  54. Winfree R, Williams NM, Dushoff J, and Kremen C. 2007. Native bees provide insurance against ongoing honey bee losses. Ecol. Lett 10: 1105–1113. [DOI] [PubMed] [Google Scholar]

RESOURCES