Skip to main content
Elsevier Sponsored Documents logoLink to Elsevier Sponsored Documents
. 2014 Mar;103(100):129–137. doi: 10.1016/j.beproc.2013.11.013

Familiarity with the experimenter influences the performance of Common ravens (Corvus corax) and Carrion crows (Corvus corone corone) in cognitive tasks

Lara Cibulski a,b,c,, Claudia AF Wascher a,b,1, Brigitte M Weiß a,b,d,2, Kurt Kotrschal a,b,3
PMCID: PMC4003535  PMID: 24333226

Highlights

  • We compared the results of corvids in experiments with familiar/unfamiliar humans.

  • We investigated behavioural reactions towards familiar and unfamiliar humans.

  • Corvids performed significantly better in experiments with familiar humans.

  • Corvids did not show more neophobia towards unfamiliar humans.

  • Hence, familiarity positively affected the experimental performance of corvids.

Keywords: Carrion crow, Cognition, Common raven, Familiarity, Human–animal relationships

Abstract

When humans and animals interact with one another over an extended time span they familiarise and may develop a relationship, which can exert an influence on both partners. For example, the behaviour of an animal in experiments may be affected by its relationship to the human experimenter. However, few studies have systematically examined the impact of human–animal relationships on experimental results. In the present study we investigated if familiarity with a human experimenter influences the performance of Common ravens (Corvus corax) and Carrion crows (Corvus corone corone) in interactive tasks. Birds were tested in two interactive cognitive tasks (exchange, object choice) by several experimenters representing different levels of familiarity (long and short-term). Our findings show that the birds participated more often in both tasks and were more successful in the exchange task when working with long-term experimenters than when working with short-term experimenters. Behavioural observations indicate that anxiety did not inhibit experimental performance but that the birds’ motivation to work differed between the two kinds of experimenters, familiar and less familiar. We conclude that human–animal relationships (i.e. familiarity) may affect the experimental performance of corvids in interactive cognitive tasks.

1. Introduction

Human–animal relationships can have a strong emotional component similar to relationships between humans (Kotrschal et al., 2009; McNicholas et al., 2005). Companion animals may actually contribute to the well-being and even health of humans, for example by providing social support and by facilitating social contacts to other humans (Beetz et al., 2012; McNicholas et al., 2005; Prothmann, 2008). Likewise, humans considerably influence companion animals: stroking, for instance, can reduce acute stress responses in dogs (Hennessy et al., 1998). Kotrschal et al. (2009) found effects of the owner–dog relationship on stress hormone levels and working performance of human–dog dyads: owners, which were closely attached to their dogs, effectively provided social support to their dogs in stressful situations. Such effects do not seem to be limited to domesticated species. In Greylag geese (Anser anser), for instance, it has been shown that human foster parents provide social support as effectively as goose parents (Frigerio et al., 2003; Weiß and Kotrschal, 2004).

Human–animal relationships may develop whenever humans and animals are able to distinguish each other individually and interact with each other regularly, for example during care-taking activities (Bayne, 2002; Chang and Hart, 2002; Russow, 2002). The ability to recognise individual humans has already been shown in a variety of species including several species of corvids (pigeons, Columba livia: Belguermi et al., 2011; Common ravens, Corvus corax: Bugnyar, 2007; review on mammal and bird species: Davis, 2002; magpies, Pica pica: Lee et al., 2011; mockingbirds, Mimus polyglottos: Levey et al., 2009; American crows, Corvus brachyrhynchos: Marzluff et al., 2010; horses, Equus caballus: Sankey et al., 2010; Carrion crows, Corvus corone corone: Wascher et al., 2012). Furthermore, it is known that the neurology (Goodson, 2005) and physiology (DeVries et al., 2003) of social behaviour are highly conserved among vertebrates and that the same mechanisms for bonding are involved in humans, and some animals (Odendaal, 2000). During the familiarisation with individual humans stress and neophobia of animals are gradually reduced (Bayne, 2002; Chang and Hart, 2002; Russow, 2002). In agreement with this, von Bayern and Emery (2009) showed that jackdaws (Corvus monedula) took longer to retrieve hidden food in the presence of an unfamiliar human than in the presence of a familiar person. The reduction of neophobia may also increase the animals’ motivation to work and thereby facilitate experimental procedures (Davis, 2002). Therefore, the presence of individual humans (Bayne, 2002; Davis, 2002) or interactions between humans and animals (Odendaal, 2000) may ultimately influence scientific results, such as measurements of hormone concentrations or anxiety-like behaviours. It has even been assumed that human–animal relationships possibly affect the performance of non-human animals in cognitive tasks and that a positive relationship between the experimenter and the focal individual may have contributed to the discovery of impressive cognitive abilities - for example by Irene Pepperberg in Grey parrots (Davis, 2002). Recent evidence suggests that human–animal relationships may indeed play a role during certain experiments (Péron et al., 2012). While there may be nothing wrong with optimising test performance by maintaining optimal relationships with an experimental animal (Péron et al., 2012), the problem may rather be that animals may not cooperate with unfamiliar experimenters, thus showing suboptimal performance that would wrongly be interpreted as evidence for cognitive constraints in a certain individual/species. On the contrary, concerns have been expressed that human–animal relationships may interfere with experimenter objectivity (Schilhab, 2002) and could lead to “Clever Hans” phenomena (Miklósi and Soproni, 2005; Rosenthal, 1967).

Recent research has revealed impressive cognitive abilities in various corvid species (rooks, Corvus frugilegus: Bird and Emery, 2009; Common ravens, C. corax: Bugnyar, 2007; Western scrub-jays, Aphelocoma californica: Dally et al., 2004; New caledonian crows, Corvus moneduloides: Hunt, 1996; review on several corvid species: Seed et al., 2009). A number of these studies made use of interactive experiments, that is experiments which necessitate contact between the bird and the experimenter or which involve manipulations by the experimenter (Common ravens, C. corax and Carrion crows, C. corone corone: Dufour et al., 2012; Carrion crows, C. corone corone: Mikolasch et al., 2011; jackdaws, C. monedula: Schloegl, 2011; Clark's nutcrackers, Nucifraga columbiana: Tornick et al., 2010; Carrion crows, C. corone corone: Hoffmann et al., 2011; jackdaws, C. monedula: von Bayern and Emery, 2009). One of the reasons why working with corvids is challenging is their pronounced neophobia (Heinrich, 1988, 1999; Heinrich et al., 1995). Corvids are proven to be able to distinguish between individual humans (Common ravens, C. corax: Bugnyar et al., 2007; American crows, C. brachyrhynchos: Marzluff et al., 2010; Magpies, Pica pica: Lee et al., 2011; Carrion crows, C. corone corone: Wascher et al., 2012) and also show neophobic behaviour towards unfamiliar humans (Heinrich, 1999; von Bayern and Emery, 2009). Thus, interactive experiments with corvids require a certain familiarity with the experimenter. However, the impact of familiarity or human–animal relationships on the results of corvid cognition studies has never been investigated systematically. Therefore, the aim of our study was to provide insight into the effects of familiarity with the experimenter on corvid cognition research. Specifically, we tested the hypothesis that familiarity influences the behavioural response of corvids to experimenters and the experimental performance in interactive cognitive tasks. Several different experimenters representing different levels of familiarity conducted two interactive experiments, an exchange and an object choice task, with Common ravens (C. corax) and Carrion crows (C. corone corone). We predicted that the birds’ participation rates and performance would be positively affected by familiarity with the experimenter. In addition to the experiments, the behavioural reactions of the birds to the different experimenters were monitored. We expected that the animals would show more affiliative and less stress-related behaviours towards more familiar experimenters.

2. Methods

2.1. Study subjects

The study was conducted with five captive Common ravens (three males, two females; age 2–15) and seven captive Carrion crows (three males, four females; age 2–4) at the Konrad Lorenz Research Station (KLF) in Grünau im Almtal (Austria) between January and June 2011.

Birds were kept in outdoor aviaries and could be individually distinguished by coloured leg bands. Crows were kept as two pairs and one trio; ravens were kept as two pairs and two singles which were given the opportunity to pair during the course of the study. All birds were fed a mixed diet (meat, bread, fruit, vegetables, milk products) twice daily and water was available ad libitum for drinking and bathing. Birds were not food-deprived prior to the experiments. Except for one raven (a zoo-bred individual) and one crow (a wild bird that was injured and delivered to a shelter shortly after fledging) all birds were hand-raised. All birds regularly participated in different studies investigating their cognitive abilities. The birds were separated for the tests so that they could be tested individually. However, they always had visual and acoustic contact to the other bird(s).

2.2. Experimenters

In total, 12 experimenters participated in the present study (termed experimenters A–L in the following). All were female to avoid possible effects of sex differences in interaction style (Hennessy et al., 1998; Kotrschal et al., 2009; Wedl et al., 2010). Experimenters who had performed experiments with the birds and carried out feeding duties for at least two months were considered “long-term experimenters” (experimenters A–E). Note that experimenter D was a long-term experimenter for the ravens and a short-term experimenter for the crows as she had only worked with the ravens prior to this study. Long-term experimenters did not go through a special preparation phase prior to the experiments. In contrast, “short-term” experimenters were newly introduced to the birds for the present study (experimenters D, F, G, H, I, J, K, L). They had a seven-day habituation period with the birds and then performed experiments for four days. For the habituation the short-term experimenters approached the aviaries twice daily for approximately 5–10 min and moved around in front of the wire mesh.

Various studies indicate that clothes are not crucial for human individual recognition in birds (Belguermi et al., 2011; Lee et al., 2011; Levey et al., 2009; Marzluff et al., 2010). Nonetheless, to exclude any potential effects of different clothing (Heinrich, 1999; Rybarczyk et al., 2003) all experimenters wore an identical shirt and jeans during habituation and experiments. For the experiments all experimenters announced themselves saying “hallo” to the birds but subsequently conducted the experiments without talking to the birds any more. Due to logistical constraints some experimenters worked only with the crows, and testing with long and short-term experimenters could not be balanced across seasons (Table 1).

Table 1.

Testing schedules for Common ravens and Carrion crows. The level of familiarity (short-/long-term) is given for each experimenter.

Pre-breeding
Breeding
Post-breeding
January February March April May June
Crows
Familiarity Long Short Short Long Short Long
Experimenter A, B, C D, F G, H E I, J, K, L C



Ravens
Familiarity Long Short Long Short Long
Experimenter C, D G, H B, E I, J, K, L C

2.3. Experimental procedure

In both tasks, the birds performed two sessions of 10 trials each with each experimenter in pseudo-randomised order (note that experimenters worked during different seasons and experimenter order was thus not fully randomised; see Table 1). Half of the birds started with the exchange task and the other half started with the object choice task. Experiments were conducted in the morning (between 0800 h and 1200 h) and in the afternoon (between 1400 h and 1800 h). The birds were separated from their conspecifics prior to the experiment by LC whereas only the respective experimenter was present during the experiments. The experiments were conducted through the wire mesh (i.e. the experimenter was outside the aviary). The experiments were video-recorded and analysed by LC (participation: yes/no, performance: correct/incorrect). In addition, long-term experimenters noted the results on data sheets during the experiment to check for interobserver-reliability (accordance of the results from data sheets and video coding: 100%).

2.3.1. Exchange task

Ravens and crows were previously trained to exchange non-preferred food for preferred food items (Dufour et al., 2012). This task involves a cost-benefit consideration by the bird which is presumably affected by the subject's motivation and the effort necessary to obtain the most valuable food item (Dufour et al., 2012). In a similar delayed gratification task children considered experimenter reliability when deciding whether to consume a small reward instantly or wait for a larger reward (Kidd et al., 2013). Thus, the birds’ behaviour in the exchange task could also be affected by whether they consider the experimenter to be a reliable exchange partner.

One piece of standard food (bread, approximately 1.0 cm × 1.0 cm × 0.7 cm for ravens, 0.7 cm × 0.7 cm × 0.5 cm for crows) and one piece of same size preferred food (cheese) were presented in the palms of the experimenter out of reach of the bird. The experimenter then passed the standard food to the bird. After the bird had taken the standard food, the experimenter waited for two seconds with the presenting hand closed to a fist to allow the bird to make a decision. The preferred food item remained visible in the other palm. Then the experimenter opened the fist so that the bird could return the standard food into the opened palm. If the bird returned the standard food it received the preferred food item. A trial was considered successful only if the bird took the standard food and returned it to the experimenter as described. An interruption of this process by eating or caching the standard food was considered a failure. If a bird did not take the standard food or if it did not approach the experimenter so that an interaction was not possible, this was considered non-participation and the experimenter continued with the next trial.

2.3.2. Object choice task

In the object choice task the birds were given a cue by the experimenter to indicate food hidden in one of two cups. The study subjects had previously participated successfully in such tasks (Mikolasch et al., 2011). The experimenter knelt in front of the wire mesh outside the aviary. A piece of food (1/16 piece of commercial dog food for crows and 1/8 for ravens) was hidden below one of two red cups (1 cm high, diameter 4 cm) on a blue board (50 cm × 20 cm). Cups were placed approximately 25 cm apart. The food was hidden under the cups behind a blind (35 cm × 16 cm × 12 cm), out of view of the birds. The position of the reward (left/right) was randomised and both cups were rewarded equally often. After removing the blind the experimenter provided the bird with a cue about the location of the food by turning her head to look at the rewarded cup and touching it three times with the ipsilateral hand. Afterwards, the experimenter took her hand back, held it in a relaxed position close to her body and looked at the bird. After three seconds the board was moved towards the bird to allow it to choose a cup by touching it with the beak. Then the board was moved away from the bird again and the cups were turned over so that the bird could see under which cup the reward was hidden. If the choice was correct the bird received the food item. In case of a wrong choice the reward was placed in the middle of the blue board and the next trial was started. To attract a focal individual's attention in case it did not approach the setup or did not choose, the experimenter put her hands to the edges of the board, called the bird's name and placed her hands back towards her body. If the bird still did not approach or make a choice, this sequence was repeated. If the bird did not react to this twice, this was considered non-participation.

2.4. Behavioural observations

The behaviour of the birds towards the different experimenters was monitored outside of the experimental context. The experimenters approached the aviaries (in case of short-term experimenters after seven days of habituation) twice and moved along the aviary slowly at a distance of approximately 0.5 m to the wire mesh. The behaviour of the birds was video-recorded by LC for 5 min per approach and videos were coded with Solomon Coder beta 11.07.04 (Copyright by András Péter; http://solomoncoder.com) by LC. Special attention was paid to affiliative (“approach”), stress-related (“wing-quivering”, “fluttering”) and comfort behaviours (“preen”, “pluster”, “stretch”, “scratch”; Table S1, see supplementary data).

2.5. Statistical analysis

We performed generalised linear mixed models (GLMMs) with SPSS 19 for the dependent variables “participation” (yes/no) and “performance” (correct/incorrect) using a binomial logistic regression. In addition to “familiarity” we considered all factors which we think could have contributed to differences in participation and performance rates: the identity of each “person” (nested within familiarity) was included to investigate if potential differences were caused by individual-specific characteristics of the experimenter (by individual-specific characteristics we mean, e.g. body language, voice, hair colour, etc.). “Season” (divided into pre-breeding, breeding and post-breeding season, see Table 1) was included to check if seasonal effects, which are known to strongly affect bird behaviour (Helm et al., 2006; Nelson, 2005), contributed to potential differences. In addition the interactions species × familiarity and season × familiarity were included due to the unbalanced study design. “Animal identity” was included as a random term in all models to account for repeated measurements.

Due to the unbalanced participation of the different experimenters over the course of the study, the effects of “person” and “season” were confounded. Therefore, two separate analyses were performed containing either “season” (S models) or “person” (P models) as a fixed factor for each dependent variable. These models were compared by the Akaike Information Criterion corrected for small sample sizes (AICc), a measure of model accuracy (Garamszegi et al., 2009), whereby an AICc that is smaller by two or more units indicates a better model fit. The results of the parallel S and P models were similar: in most cases terms were significant in both models and the magnitude and direction of differences were similar. Since the S models had lower AICc values, tables and graphs in the following section show the results of S models complemented by additional results from P models.

We reduced the full models by stepwise exclusion of non-significant fixed terms, taking into consideration the AICc: if the exclusion of a non-significant term increased the AICc this term was re-entered into the model. Excluded terms were re-entered into the final model singly to confirm non-significance (Garamszegi et al., 2009). Pairwise comparisons were calculated for fixed factors with more than two factor levels and the method of least significant differences was used for post hoc corrections. Estimated mean values (EM) and standard errors (SE) are given for the factor levels. Estimated means differ from actual means in that they are calculated taking into consideration all factors included in the model. The advantage of this method is that the effects of multiple factors on the variable of interest are not treated singly but are considered simultaneously. We present estimated means rather than actual means to integrate all investigated factors in this manner. To assess the relative importance of significant fixed effects, effect sizes were taken into account: effects with effect sizes higher than one were considered strong effects, effect sizes of about 0.5 moderate and effect sizes below 0.5 were considered weak effects (Garamszegi et al., 2009).

In the object choice task both cups were rewarded equally often, so that the birds could choose correctly in 50% of the trials even if they chose cups randomly. Therefore we performed binomial tests on the performances of each bird with the different experimenters to test if the birds’ performance deviated from a chance level of 50% or 0.5, respectively. Additionally we calculated the percentage of correct trials for each bird and evaluated differences in performance between long and short-term experimenters using Wilcoxon signed-rank tests.

For the statistical analysis of behavioural observations only behaviours occurring in more than 10% of the approaches were used. The results were then transformed into binomial data (occurrence of the behaviour yes/no), summarised according to functional contexts (Table S1, see supplementary data) and analysed with GLMMs with the same parameters as the experimental results. For brevity's sake we only report significant terms that remained in the respective final model in the results. Full results are given in the tables (Tables 2–4).

Table 2.

Results of GLMMs with participation in the exchange task as the dependent variable. Statistical parameters for the final model are given as well as factors entered into the model, degrees of freedom (df), F- and p-values. F- and p-values of excluded factors were taken from full models. Effects sizes are given for terms that remained in the final model. The results shown originate from S models; results written in italics originate from models containing the fixed factor “person”.

Factor df F p Factor level Effect size
Final model 7 25.168 <0.001



Species 1 0.833 0.362 Crow 0.135
Raven 1.590



Familiarity 1 43.350 <0.001 Long-term −0.458
Short-term 0



Season 2 24.933 <0.001 Pre-breeding −0.132
Breeding 0.284
Post-breeding 0



Species × familiarity 1 5.236 0.022 Crow × long-term 1.101
Raven × long-term 0
Raven × short-term 0



Season × familiarity 2 22.604 <0.001 Pre-breeding × long-term 2.259
Breeding × long-term 2.214
Post-breeding × long-term 0
Pre-breeding × short-term 0
Breeding × short-term 0
Post-breeding × short-term 0



Person (familiarity) 11 9.971 <0.001 A 6.780
B 6.628
C −2.171
D 4.680
E 0
D −1.785
F −2.215
G −2.339
H −0.484
I −2.294
J −1.980
K −2.679
L 0

Table 3.

Results of GLMMs with performance in the exchange task as the dependent variable. Statistical parameters for the final model are given as well as factors entered into the model, degrees of freedom (df), F- and p-values. F- and p-values of excluded factors were taken from full models. Effects sizes are given for terms that remained in the final model. The results shown originate from S models; results written in italics originate from models containing the fixed factor “person”.

Factor df F p Factor level Effect size
Final model 6 29.201 <0.001



Species 1 5.607 0.018 Crow −0.904
Raven 2.765



Familiarity 1 73.751 <0.001 Long-term 2.524
Short-term 0



Season 2 52.590 <0.001 Pre-breeding 1.285
Breeding −1.192
Post-breeding 0



Species × familiarity 2 2.857 0.058 Crow × long-tern −3.108
Raven × long-term 0
Crow × short-term −3.877
Raven × short-term 0



Season × familiarity 2 9.218 <0.001 Pre-breeding × long-term −1.918
Breeding × long-term −1.139
Post-breeding × long-term 0
Pre-breeding × short-term 0
Breeding × short-term 0
Post-breeding × short-term 0



Person (familiarity) 11 11.100 <0.001 A 2.282
B 1.171
C 1.744
D −0.025
E 0
D 1.743
F 0.861
G −1.646
H −0.948
I 0.175
J 0.030
K −0.337
L 0

Table 4.

Results of GLMMs with participation in the object choice task as the dependent variable. Statistical parameters for the final model are given as well as factors entered into the model, degrees of freedom (df), F- and p-values. F- and p-values of excluded factors were taken from full models. Effects sizes are given for terms that remained in the final model. The results shown originate from S models; results written in italics originate from models containing the fixed factor “person”.

Factor df F p Factor level Effect size
Final model 7 42.482 <0.001



Species 1 2.043 0.153 Crow 1.421
Raven −0.348



Familiarity 1 80.397 <0.001 Long-term −0.954
Short-term 0



Season 2 72.846 <0.001 Pre-breeding −1.765
Breeding 0.976
Post-breeding 0



Species × familiarity 1 3.961 0.047 Crow × long-term 0.745
Crow × short-term 0
Raven × long-term 0
Raven × short-term 0



Season × familiarity 2 49.158 <0.001 Pre-breeding × long-term 2.958
Breeding × long-term 2.568
Post-breeding × long-term 0
Pre-breeding × short-term 0
Breeding × short-term 0
Post-breeding × short-term 0



Person (familiarity) 11 17.241 <0.001 A 2.384
B −0.421
C −0.812
D 1.972
E 0
D −2.966
F −1.955
G 0.148
H 0.349
I −0.749
J −0.824
K −0.972
L 0

3. Results

3.1. Exchange task

Participation rates in the exchange task were significantly affected by familiarity: the birds participated more often in the experiments with long-term experimenters (EMtrials participated = 0.974 ± 0.013) than with short-term experimenters (EMtrials participated = 0.886 ± 0.050; Fig. 1A). Seasonal effects on participation rates were weak; the birds participated more often during the pre-breeding (EMtrials participated = 0.954 ± 0.022) and breeding (EMtrials participated = 0.969 ± 0.016) season than during the post-breeding season (EMtrials participated = 0.885 ± 0.051). The interaction species × familiarity remained in the final model, but pairwise comparisons did not show any significant results. Participation rates were higher with long-term experimenters during the pre-breeding (EMtrials participated = 0.985 ± 0.008) and breeding season (EMtrials participated = 0.990 ± 0.006) than during the post-breeding season (EMtrials participated = 0.890 ± 0.053). The birds participated more often in the task with long-term than with short-term experimenters during the pre-breeding (long-term experimenters: EMtrials participated = 0.985 ± 0.008; short-term experimenters: EMtrials participated = 0.866 ± 0.061) and breeding season (long-term experimenters: EMtrials participated = 0.990 ± 0.006; short-term experimenters: EMtrials participated = 0.907 ± 0.043). Person-specific effects were strong but not significant in pairwise comparisons (GLMM, Table 2). The lower AICc of the S model (12,165.907 as compared to an AICc of 16,187.614 in the P model) indicates that seasonal effects explain the observed differences better than person-specific effects.

Fig. 1.

Fig. 1

Estimated mean participation rates (A) and performance (B) ±SE in the exchange task with long-term -and short-term experimenters (“season” model data). ***p < 0.001.

There was also a strong effect of familiarity with the experimenter on performance: the birds exchanged food successfully more often in experiments with long-term (EMcorrect exchanges = 0.831 ± 0.110) than with short-term experimenters (EMcorrect exchanges = 0.521 ± 0.194; Fig. 1B). Seasonal effects on performance were strong and significant in pairwise comparisons: the birds performed significantly better during the pre-breeding (EMcorrect exchanges = 0.838 ± 0.107) and the post-breeding (EMcorrect exchanges = 0.789 ± 0.132) than during the breeding season (EMcorrect exchanges = 0.391 ± 0.185). Ravens (EMcorrect exchanges = 0.935 ± 0.072) performed significantly better than crows (EMcorrect exchanges = 0.270 ± 0.197). Performance with long-term experimenters was higher during the pre-breeding (EMcorrect exchanges = 0.875 ± 0.087) and post-breeding (EMcorrect exchanges = 0.929 ± 0.055) than during the breeding season (EMcorrect exchanges = 0.562 ± 0.195). Performance with short-term experimenters was significantly higher during the pre-breeding (EMcorrect exchanges = 0.792 ± 0.132) than during the breeding (EMcorrect exchanges = 0.243 ± 0.144) and post-breeding season (EMcorrect exchanges = 0.514 ± 0.195). The birds performed worse with short-term experimenters during the breeding than during the post-breeding season. Significant differences between long-term and short term experimenters concerning performance occurred during the breeding (long-term experimenters: EMcorrect exchanges = 0.562 ± 0.195; short-term experimenters: EMcorrect exchanges = 0.243 ± 0.144) and post-breeding season (long-term experimenters: EMcorrect exchanges = 0.929 ± 0.055; short-term experimenters: EMcorrect exchanges = 0.514 ± 0.195) but not during the pre-breeding season (long-term experimenters: EMcorrect exchanges = 0.875 ± 0.087; short-term experimenters: EMcorrect exchanges = 0.792 ± 0.132), although the effect of familiarity was strongest during the pre-breeding season (GLMM, Table 3). Person-specific effects were strong and pairwise comparisons showed that the performance of the birds differed significantly between different experimenters (GLMM, Table 3). Again the AICc of the S model (AICc = 9221.836) was lower than the AICc of the P model (AICc = 9256.926), indicating a better fit of the S model.

3.2. Object choice task

Familiarity had a strong effect on the birds’ participation rates: the birds participated more often in the object choice task when they were working with long-term experimenters (EMtrials participated = 0.865 ± 0.088) than when they were working with short-term experimenters (EMtrials participated = 0.644 ± 0.171; Fig. 2A). Season also had a strong effect, which remained significant in pairwise comparison: participation in the object choice task was significantly higher during the breeding (EMtrials participated = 0.944 ± 0.040) than during the pre-breeding season (EMtrials participated = 0.569 ± 0.185) and the post-breeding season (EMtrials participated = 0.638 ± 0.174). Ravens participated with long-term experimenters more often (EMtrials participated = 0.645 ± 0.275) than with short-term experimenters (EMtrials participated = 0.428 ± 0.291). Participation rates with long term experimenters were significantly higher during the pre-breeding (EMtrials participated = 0.813 ± 0.118) and breeding season (EMtrials participated = 0.978 ± 0.016) than during the post-breeding season (EMtrials participated = 0.568 ± 0.189). The birds participated significantly more often with short-term experimenters during the breeding (EMtrials participated = 0.862 ± 0.090) and post-breeding season (EMtrials participated = 0.702 ± 0.157) than during the pre-breeding season (EMtrials participated = 0.287 ± 0.157); participation rates with short-term experimenters were significantly higher during the breeding than during the post-breeding season. Participation rates with long-term experimenters and short-term experimenters differed significantly during the pre-breeding (long-term experimenters: EMcorrect exchanges = 0.813 ± 0.118; short-term experimenters: EMcorrect exchanges = 0.287 ± 0.157) and post-breeding season (long-term experimenters: EMcorrect exchanges = 0.568 ± 0.189; short-term experimenters: EMcorrect exchanges = 0.702 ± 0.157), although the effect of familiarity was strong during the breeding season as well (GLMM, Table 4). Person-specific effects were strong and pairwise comparisons showed that the participation rates of the birds differed significantly between different experimenters (GLMM, Table 4). Model fit of the S model (AICc = 14,194.300) was better than model fit of the P model (AICc = 14,240.664).

Fig. 2.

Fig. 2

Estimated mean participation rates (A) and performance (B) ±SE in the object choice task with long-term and short-term experimenters (“season” model data). ***p < 0.001.

None of the fixed terms had a significant effect on performance in the object choice task (GLMM, Table S2, see supplementary data, Fig. 2B). A further analysis of performance revealed that only in a few cases the birds’ performance differed from chance level at all. Two crows performed above chance level with short-term experimenter G (binomial test, p = 0.002 and p = 0.0041, respectively). One crow performed significantly below chance level with short-term experimenter J (binomial test, p = 0.041). On a group level the birds’ performance did not differ from chance level with either long-term (Wilcoxon signed-rank test, n = 5, Z = −1.557, p = 0.120, Median 52.94% of trials correct) or short-term experimenters (Wilcoxon signed-rank test, n = 8, T = −0.178, p = 0.859, Median 49.38% of trials correct).

3.3. Behavioural observations

Concerning the birds’ behaviour towards the experimenters we found that only the frequency of approaches towards the experimenter differed significantly between long-term and short-term experimenters (GLMM, Table S3, see supplementary data): the birds approached long-term experimenters (EMoccurence = 0.304 ± 0.135) significantly more often than short-term experimenters (EMoccurence = 0.062 ± 0.034). There was a significant interaction of season and familiarity with long-term experimenters being approached less often during the pre-breeding (EMoccurence = 0.071 ± 0.060) than during the breeding season (EMoccurence = 0.713 ± 0.142). Long-term experimenters (EMoccurence = 0.713 ± 0.142) were approached significantly more often than short-term experimenters (EMoccurence = 0.034 ± 0.026) during the breeding season. The P model revealed that ravens (EMoccurence = 0.469 ± 0.165) approached experimenters significantly more often than crows (EMoccurence = 0.081 ± 0.044). Ravens (EMoccurence = 0.809 ± 0.132) tended to approach long-term experimenters more often than crows (EMoccurence = 0.139 ± 0.082). Ravens also tended to approach long-term experimenters more often than short-term experimenters (EMoccurence = 0.156 ± 0.089). The AICc of the P model was lower (AICc = 1168.727) than the AICc of the S model (AICc = 1208.150).

Ravens (EMoccurence = 0.692 ± 0.068) showed more comfort behaviour than crows (EMoccurence = 0.251 ± 0.046) and the occurrence of comfort behaviours differed between seasons: frequencies of comfort behaviour were significantly higher during the pre-breeding (EMoccurence = 0.625 ± 0.081) than during the breeding (EMoccurence = 0.382 ± 0.069) and post-breeding season (EMoccurence = 0.388 ± 0.062). Model fit of the P model (AICc = 984.934) was better than of the S model (AICc = 992.420). Familiarity, however, had no effect on the occurrence of comfort behaviours (GLMM, Tab. S4, see supplementary data). None of the fixed terms had a significant effect on the occurrence of stress-related behaviours (GLMM, Tab. S5, see supplementary data).

4. Discussion

For the first time we demonstrate effects of familiarity on the performance and behaviour of corvids in interactive cognitive tasks: birds participated more often in an exchange and in an object choice task when working with a long-term experimenter than when working with a short-term experimenter. In addition, the birds’ success rates in the exchange task were higher when working with long-term experimenters. We thereby provide evidence that familiarity may not only affect anxiety-like or stress-related behaviours as previously reported (von Bayern and Emery, 2009; Bayne, 2002; Chang and Hart, 2002) but also the outcome of interactive cognitive experiments. Success rates in the object choice task were not affected by familiarity with the experimenter.

During behavioural observations the birds did not show more stress-related behaviours towards short-term than towards long-term experimenters. This indicates that, unlike in other studies (von Bayern and Emery, 2009), the birds’ behaviour during the experiments is unlikely to be affected by the corvid-typical neophobia towards unfamiliar humans. These findings agree with other reports of neophobia reduction in the course of repeated interactions between humans and animals (Bayne, 2002; Chang and Hart, 2002; Russow, 2002). Therefore we assume, that the birds had habituated to short-term experimenters even within the comparably short time-span of one week, although we cannot completely exclude the possibility that the birds still were more nervous in the presence of short-term experimenters. Thus, we suggest that differences in the birds’ motivation to work, but not neophobia as described by von Bayern and Emery (2009), may account for the birds’ differential participation rates and performances when working with long or short-term experimenters.

Considering recent evidence for children's sensitivity towards experimenter reliability in a delayed gratification task (Kidd et al., 2013) it can be assumed that our subjects perceived long-term experimenters as more reliable exchange partners, either due to their shared experimental history or an existing human–animal-relationship: the birds did not only participate more often in both tasks and performed better in the exchange task but also approached long-term experimenters more often than short-term experimenters during behavioural observations. Comparable observations were made in studies on horses and were interpreted as an indication for the existence of a relationship between the animals and familiar experimenters (Sankey et al., 2010). In a review of his work Davis (2002) reports that even brief sociopositive interactions with a handler lead to a preference for contact with this handler in rats. These studies indicate that animals may develop preferences for familiar humans (Davis, 2002). We suggest that long-term experimenters and birds shared a relationship that affected the birds’ behaviour during the experiments, which is supported by the levels of affiliative behaviour the birds displayed towards long-term experimenters (Sankey et al., 2010). However, our results may also have been caused by an intermixture of the effects of familiarity and reinforcement history: long-term experimenters had regularly shared positive interactions with the birds, fed them and provided rewards during experiments in the past. This might have caused the birds to develop a preference for interactions with long-term experimenters (Davis, 2002). On the other hand, repeated positive interactions between humans and animals are known to promote the formation of human–animal relationships (Bayne, 2002; Chang and Hart, 2002; Russow, 2002). Currently, it is impossible to determine if our findings were caused by reinforcement history or by a relationship between the subjects and long-term experimenters. Therefore, in the future, the effect of reinforcement on the development of preferences and human–animal relationships needs to be investigated. Also, means to measure human–animal relationships, for example by quantifying affiliative behaviours and human–animal interactions (Sankey et al., 2010), might be useful to evaluate the relationship between individual experimenters and their subjects more precisely.

Notwithstanding, our findings demonstrate that familiarity may have considerable effects on the results of interactive cognitive experiments with corvids and may thus be of interest for corvid cognition research in general. Moreover, we assume that our results may have consequences for interactive work with other animals: in many institutions daily care-giving is conducted by different persons than the experimental procedures and thus familiarity between experimenters and animals may be reduced or lacking entirely although its beneficial effects (e.g. reduction of animals’ stress levels) are known (Bayne, 2002; Chang and Hart, 2002; Davis, 2002). Since several species have been shown to develop preferences for familiar humans (Davis, 2002), in the future the effects of familiarity should be considered more carefully not only in corvid research but in the behavioural sciences in general.

Surprisingly, the birds’ performance in the object choice task hardly deviated from chance level at all, although previous studies have clearly demonstrated the effects of touching an object on the choice decisions of juvenile ravens and Carrion crows via local enhancement (Mikolasch et al., 2011; Schloegl et al., 2008). Also, wild ravens use gestures to communicate (Pika and Bugnyar, 2011). Possibly the birds were not able to interpret the cue and did not perceive touching the cup as object manipulation as used in the studies of Mikolasch et al. (2011) and Schloegl et al. (2008). Alternatively, the birds may not have used the cue although they had understood the intention. Currently, we cannot specify the exact reason for the birds’ failure.

The birds’ participation and performance rates differed between different seasons.

The effect of season on experimental performance was not a main focus of the present study. However, due to the experimental design we were not able to collect all data in one seasonal phase and had to perform experiments during the breeding season, during which birds generally are not very motivated to participate in cognition experiments. Surprisingly, in our experiment participation rates were lower in the pre-breeding phase than during breeding and post-breeding. However, according to our expectations, performance was lower during the breeding season than during the pre- and post-breeding season. Seasonal effects could have been caused by hormonal (e.g. elevated levels of sex steroids) and behavioural changes (e.g. territory defence, courtship) accompanying breeding (Helm et al., 2006; Nelson, 2005). Ravens participated more often in the exchange task and were more successful in this task than crows; in contrast, ravens participated less often in the object choice task. The effect of familiarity with the experimenter, however, was not influenced by the species. Hence, neither species nor the unbalanced distribution of species across seasons can account for the present results. We also detected person-specific effects on participation rates and performance. Person-specific effects on human–animal-interactions were shown in a study on dogs (Kotrschal et al., 2009). These effects are assumed to be mediated by the owner's interaction style, which is affected by owner personality (Kotrschal et al., 2009). However, due to logistical reasons we could not take into account the experimenters’ personality or interaction style in the present study. Therefore, in the future it might be interesting to examine potential effects of these factors and to systematically vary the experimenters’ interaction style. Although effects of season and person were confounded to some extent and further studies will be needed to determine the importance of seasonal and experimenter-specific factors on corvids’ behaviour in interactive cognitive tasks, the effects of familiarity with the experimenter on participation rates and performance were robust.

In summary we show that the effects of familiarity between an experimenter and her subjects extend beyond a reduction of stress- and anxiety-like behaviours (Bayne, 2002; Davis, 2002) and that familiarity even affects the outcome of interactive cognitive experiments. In the future, these effects should be examined and considered more carefully not only in corvid research but in the behavioural sciences in general, as many species are theoretically capable of forming relationships with humans, which, in turn, can influence experimental results.

Acknowledgements

We acknowledge funding by the Austrian Science Fund (FWF) project P20538-B17, the Verein der Förderer der KLF and the Cumberland Stiftung. Special thanks go to G. Szipl, A. Braun, S. Singh-Sondhi, M. Spreafico, M. Klumb, C. Herrmann, C. Swaton, C. Köck, L. Nowack and V. Ziemen for help with the experiments, to C. Schloegl for help with the statistical analyses and to C. Rockenbach and K. Bengelsdorf for proofreading the manuscript. We thank D. Kelly and three anonymous reviewers for valuable comments on an earlier version of the manuscript.

Footnotes

This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Contributor Information

Lara Cibulski, Email: lara.cibulski@hotmail.de.

Claudia A.F. Wascher, Email: claudia.wascher@klf.ac.at.

Brigitte M. Weiß, Email: brigitte.weiss@klf.ac.at.

Kurt Kotrschal, Email: kurt.kotrschal@klf.ac.at.

Appendix A. Supplementary data

The following are the supplementary data to this article:

mmc1.docx (23.2KB, docx)

References

  1. Bayne K. Development of the human-research animal bond and its impact on animal well-being. ILAR Journal. 2002;43:4–9. doi: 10.1093/ilar.43.1.4. [DOI] [PubMed] [Google Scholar]
  2. Beetz A., Uvnäs-Moberg K., Julius H., Kotrschal K. Psychosocial and psychophysiological effects of human–animal interactions: the possible role of oxytocin. Frontiers in Psychology. 2012;3 doi: 10.3389/fpsyg.2012.00234. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Belguermi A., Bovet D., Pascal A., Prévot-Julliard A.-C., Saint Jalme M., Rat-Fischer L., Leboucher G. Pigeons discriminate between human feeders. Animal Cognition. 2011;14:909–914. doi: 10.1007/s10071-011-0420-7. [DOI] [PubMed] [Google Scholar]
  4. Bird C.D., Emery N.J. Rooks perceive support relations similar to six-month-old babies. Proceedings of the Royal Society B: Biological Sciences. 2009;277:147–151. doi: 10.1098/rspb.2009.1456. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Bugnyar T. An integrative approach to the study of ‘theory-of-mind’-like abilities in ravens. Japanese Journal of Animal Psychology. 2007;57:15–27. [Google Scholar]
  6. Bugnyar T., Schwab C., Schloegl C., Kotrschal K., Heinrich B. Ravens judge competitors through experience with play caching. Current Biology. 2007;17:1804–1808. doi: 10.1016/j.cub.2007.09.048. [DOI] [PubMed] [Google Scholar]
  7. Chang F.T., Hart L.A. Human–animal bonds in the laboratory: how animal behavior affects the perspective of caregivers. ILAR Journal. 2002;43:10–18. doi: 10.1093/ilar.43.1.10. [DOI] [PubMed] [Google Scholar]
  8. Dally J.M., Emery N.J., Clayton N.S. Cache protection strategies by western scrub-jays (Aphelocoma californica): hiding food in the shade. Proceedings of the Royal Society B: Biological Sciences. 2004;271:S387–S390. doi: 10.1098/rsbl.2004.0190. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Davis H. Prediction and preparation: pavlovian implications of research animals discriminating among humans. ILAR Journal. 2002;43:19–26. doi: 10.1093/ilar.43.1.19. [DOI] [PubMed] [Google Scholar]
  10. DeVries A.C., Glasper E.R., Detillion C.E. Social modulation of stress responses. Physiology & Behavior. 2003;79:399–407. doi: 10.1016/s0031-9384(03)00152-5. [DOI] [PubMed] [Google Scholar]
  11. Dufour V., Wascher C.A.F., Braun A., Miller R., Bugnyar T. Corvids can decide if a future exchange is worth waiting for. Biology Letters. 2012;8:201–204. doi: 10.1098/rsbl.2011.0726. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Frigerio D., Weiss B., Dittami J., Kotrschal K. Social allies modulate corticosterone excretion and increase success in agonistic interactions in juvenile hand-raised graylag geese (Anser anser) Canadian Journal of Zoology. 2003;81:1746–1754. [Google Scholar]
  13. Garamszegi L.Z., Calhim S., Dochtermann N., Hegyi G., Hurd P.L., Jorgensen C., Kutsukake N., Lajeunesse M.J., Pollard K.A., Schielzeth H., Symonds M.R.E., Nakagawa S. Changing philosophies and tools for statistical inferences in behavioral ecology. Behavioral Ecology. 2009;20:1363–1375. [Google Scholar]
  14. Goodson J.L. The vertebrate social behavior network: evolutionary themes and variations. Hormones and Behavior. 2005;48:11–22. doi: 10.1016/j.yhbeh.2005.02.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Heinrich B. Why do ravens fear their food? Condor. 1988;90:950–952. [Google Scholar]
  16. Heinrich B. Harper Perennial; New York, London, Toronto, Sydney: 1999. Mind of the Raven: Investigations and Adventures with Wolf-Birds; p. 432pp. [Google Scholar]
  17. Heinrich B., Marzluff J., Adams W. Fear and food recognition in naive Common ravens. Auk. 1995;112:499–503. [Google Scholar]
  18. Helm B., Piersma T., van der Jeugd H. Sociable schedules: interplay between avian seasonal and social behaviour. Animal Behaviour. 2006;72:245–262. [Google Scholar]
  19. Hennessy M.B., Williams M.T., Miller D.D., Douglas C.W., Voith V.L. Influence of male and female petters on plasma cortisol and behaviour: can human interaction reduce the stress of dogs in a public animal shelter? Applied Animal Behaviour Science. 1998;61:63–77. [Google Scholar]
  20. Hoffmann A., Rüttler V., Nieder A. Ontogeny of object permanence and object tracking in the Carrion crow, Corvus corone. Animal Behaviour. 2011;82:359–367. [Google Scholar]
  21. Hunt G.R. Manufacture and use of hook-tools by New Caledonian crows. Letters to Nature. 1996;379:249–251. [Google Scholar]
  22. Kidd C., Palmeri H., Aslin R.N. Rational snacking: young children's decision-making on the marshmallow task is moderated by beliefs about environmental reliability. Cognition. 2013;126:109–114. doi: 10.1016/j.cognition.2012.08.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Kotrschal K., Schöberl I., Bauer B., Thibeaut A.-M., Wedl M. Dyadic relationships and operational performance of male and female owners and their male dogs. Behavioural Processes. 2009;81:383–391. doi: 10.1016/j.beproc.2009.04.001. [DOI] [PubMed] [Google Scholar]
  24. Lee W.Y., Lee S.-i., Choe J.C., Jablonski P.G. Wild birds recognize individual humans: experiments on magpies, Pica pica. Animal Cognition. 2011;14:817–825. doi: 10.1007/s10071-011-0415-4. [DOI] [PubMed] [Google Scholar]
  25. Levey D.J., Londono G.A., Ungvari-Martin J., Hiersoux M.R., Jankowski J.E., Poulsen J.R., Stracey C.M., Robinson S.K. Urban mockingbirds quickly learn to identify individual humans. Proceedings of the National Academy of Sciences. 2009;106:8959–8962. doi: 10.1073/pnas.0811422106. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Marzluff J.M., Walls J., Cornell H.N., Withey J.C., Craig D.P. Lasting recognition of threatening people by wild American crows. Animal Behaviour. 2010;79:699–707. [Google Scholar]
  27. McNicholas J., Gilbey A., Rennie A., Ahmedzai S., Dono J.A., Ormerod E. Pet ownership and human health: a brief review of evidence and issues. British Medical Journal. 2005;331:1252–1254. doi: 10.1136/bmj.331.7527.1252. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Miklósi Á., Soproni K. A comparative analysis of animals’ understanding of the human pointing gesture. Animal Cognition. 2005;9:81–93. doi: 10.1007/s10071-005-0008-1. [DOI] [PubMed] [Google Scholar]
  29. Mikolasch S., Kotrschal K., Schloegl C. Is caching the key to exclusion in corvids? The case of Carrion crows (Corvus corone corone) Animal Cognition. 2011;15:73–82. doi: 10.1007/s10071-011-0434-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Nelson R.J. Sinauer Associates, Inc. Publishers; Sunderland, Massachusetts: 2005. An Introduction to Behavioral Endocrinology. [Google Scholar]
  31. Odendaal J.S.J. Animal-assisted therapy – magic or medicine? Journal of Psychosomatic Research. 2000;49:275–280. doi: 10.1016/s0022-3999(00)00183-5. [DOI] [PubMed] [Google Scholar]
  32. Péron F., John M., Sapowicz S., Bovet D., Pepperberg I.M. A study of sharing and reciprocity in grey parrots (Psittacus erithacus) Animal Cognition. 2012;16:197–210. doi: 10.1007/s10071-012-0564-0. [DOI] [PubMed] [Google Scholar]
  33. Pika S., Bugnyar T. The use of referential gestures in ravens (Corvus corax) in the wild. Nature Communications. 2011;2:560. doi: 10.1038/ncomms1567. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Prothmann A. Tiergestützte Therapie – Medizin mit Streichelfaktor? Erfahrungsheilkunde. 2008;57:573–579. [Google Scholar]
  35. Rosenthal R. Covert communication in the psychological experiment. Psychological Bulletin. 1967;67:356–367. doi: 10.1037/h0024529. [DOI] [PubMed] [Google Scholar]
  36. Russow L.M. Ethical implications of the human–animal bond in the laboratory. ILAR Journal. 2002;43:33–37. doi: 10.1093/ilar.43.1.33. [DOI] [PubMed] [Google Scholar]
  37. Rybarczyk P., Rushen J., de Passillé A.M. Recognition of people by dairy calves using colour of clothing. Applied Animal Behaviour Science. 2003;81:307–319. [Google Scholar]
  38. Sankey C., Richard-Yris M.-A., Leroy H., Henry S., Hausberger M. Positive interactions lead to lasting positive memories in horses, Equus caballus. Animal Behaviour. 2010;79:869–875. [Google Scholar]
  39. Schilhab T.S.S. Anthropomorphism and mental state attribution. Animal Behaviour. 2002;63:1021–1026. [Google Scholar]
  40. Schloegl C. What you see is what you get—reloaded: can jackdaws (Corvus monedula) find hidden food through exclusion? Journal of Comparative Psychology. 2011;125:162–174. doi: 10.1037/a0023045. [DOI] [PubMed] [Google Scholar]
  41. Schloegl C., Kotrschal K., Bugnyar T. Do Common ravens (Corvus corax) rely on human or conspecific gaze cues to detect hidden food? Animal Cognition. 2008;11:231–241. doi: 10.1007/s10071-007-0105-4. [DOI] [PubMed] [Google Scholar]
  42. Seed A., Emery N., Clayton N. Intelligence in corvids and apes: a case of convergent evolution? Ethology. 2009;115:401–420. [Google Scholar]
  43. Tornick J.K., Gibson B.M., Kispert D., Wilkinson M. Clark's nutcrackers (Nucifraga columbiana) use gestures to identify the location of hidden food. Animal Cognition. 2010;14:117–125. doi: 10.1007/s10071-010-0349-2. [DOI] [PubMed] [Google Scholar]
  44. von Bayern A.M.P., Emery N.J. Jackdaws respond to human attentional states and communicative cues in different contexts. Current Biology. 2009;19:602–606. doi: 10.1016/j.cub.2009.02.062. [DOI] [PubMed] [Google Scholar]
  45. Wascher C.A.F., Szipl G., Boeckle M., Wilkinson A. You sound familiar: Carrion crows can differentiate between the calls of known and unknown heterospecifics. Animal Cognition. 2012;15:1015–1019. doi: 10.1007/s10071-012-0508-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Wedl M., Schöberl I., Bauer B., Day J., Kotrschal K. Relational factors affecting dog social attraction to human partners. Interaction Studies. 2010;11:482–503. [Google Scholar]
  47. Weiß B.M., Kotrschal K. Effects of passive social support in juvenile Greylag geese (Anser anser): a study from fledging to adulthood. Ethology. 2004;110:429–444. [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

mmc1.docx (23.2KB, docx)

RESOURCES