Object manipulation without hands

Abstract

Our current understanding of manipulation is based on primate hands, resulting in a detailed but narrow perspective of ways to handle objects. Although most other animals lack hands, they are still capable of flexible manipulation of diverse objects, including food and nest materials, and depend on dexterity in object handling to survive and reproduce. Birds, for instance, use their bills and feet to forage and build nests, while insects carry food and construct nests with their mandibles and legs. Bird bills and insect mandibles are much simpler than a primate hand, resembling simple robotic grippers. A better understanding of manipulation in these and other species would provide a broader comparative perspective on the origins of dexterity. Here we contrast data from primates, birds and insects, describing how they sense and grasp objects, and the neural architectures that control manipulation. Finally, we outline techniques for collecting comparable manipulation data from animals with diverse morphologies and describe the practical applications of studying manipulation in a wide range of species, including providing inspiration for novel designs of robotic manipulators.

1. Introduction

Manipulation (or manual dexterity) has been described as a ‘Rosetta Stone for cognition' [1], because it allows examination of cognitive abilities in naturally occurring, non-verbal behaviours. In particular, the act of grasping an object forms the foundation of manipulation, as in order to manipulate an object efficiently, an animal has to first establish a stable grip on the object. This process of grasping involves a hierarchy of control decisions: what to grasp; how to position the body and end effectors; precise continuous control to execute an action; dealing with uncertainty; detecting success or failure; and modifying subsequent actions. It also involves a close and continuous interaction between body and brain, with the mechanics of end effectors playing a significant role in the dynamics of the behaviour. Thus, precise manipulation of objects requires dynamic perception, control and adjustment of the manipulator, as grasping imposes sharp transitions in the state space of the animal's and environment's dynamics.

Having hands to manipulate objects has enabled humans to develop pivotal innovations, including tools and shelters. Consequently, biological investigation into object manipulation has been largely centred around tool use, typically those in the hands of human and non-human primates. While manipulation using forelimbs is considered a broadly shared trait in tetrapods (e.g. rats, frogs [2]), and different morphological traits in forelimbs can improve manipulative abilities [3], manipulation that occurs without hands has been largely neglected in current frameworks. The close focus on primate hands has meant that the expectation for the manipulation skills of animals without hands has been low, for instance: ‘[t]he special ability to pick things up and manipulate them … [is] something that porpoises can't do at all and crows can't do very well' [4]. After all, the etymology of the word ‘manipulate' involves ‘manus', the Latin word for hand, so it might be only natural to expect animals without hands to lack skills for manipulation. However, perhaps this anthropomorphized language may have deflected our eyes from the rich diversity of the physical interactions between animals and their environment. Once we adopt a more general definition of manipulation (e.g. ‘an agent's control of its environment through selective contact’ [5]), the absence of hands no longer means the absence of manipulation.

Indeed, manipulation skills play a significant role in the fitness of many species that lack hands. For example, a bird that fails to build a structurally sound nest might lose a safe roosting space, its potential mate, or all its eggs if the nest falls apart. Alternatively, manipulation skills may affect food-handling time, a key variable modulating energy intake rates in optimal foraging theory [6,7]. Although the ability of animals to manipulate objects underpins much of their daily lives, there is limited understanding of the underlying mechanisms that dictate successful performance, such as the morphology and sensorimotor control of the relevant appendages in animals without hands. For instance, birds typically use their bills and feet to forage and build nests, while insects such as ants, termites and wasps carry food and construct nests with their mandibles and legs. The principal appendages used (i.e. the bills in birds and the mandibles in insects) are much simpler than a hand, resembling simple robot grippers; yet they are able to accomplish highly sophisticated actions on objects in their environment.

Coincidentally, manipulation of natural objects is a major, and largely unsolved, problem for robotics. Although recent advances in gripper design [8], new sensory systems [9] and new algorithmic approaches [10], especially those that use machine learning [11], have improved performance of robotic systems in these areas, many potential robotic applications are severely limited by the inability of robots to grasp diverse objects with efficiency and reliability. It remains the case that state-of-the-art robot systems are easily outperformed by animals, including those without hands.

To initiate the integration of data and theories about manipulation with and without hands, and to inspire new avenues for designing robots that can flexibly handle objects, we focus here on manipulation in two groups of animals that achieve diverse manipulation without hands: birds and insects. For example, most of roughly 10 000 species of birds build nests (figure 1) to raise young, using materials including grass, twigs, moss, mud and even artificial objects like cigarette butts [12,13]. Similarly, insects exhibit high competence in manipulating a wide variety of objects (figure 1), to provide a shelter and food for offspring [14], to collect nectar and pollen [15], and to break up and transport large items [16–18]. We compare these two cases to the findings from primate hand research in different steps involved in grasping: the perception of objects before and during grasping, the neural basis of controlling grasp, and the act of grasping. We then discuss how recent developments in automated tracking and quantification of animal behaviour provide opportunities to move beyond simple descriptive accounts of these behaviours towards a more mechanistic understanding, and how this may provide inspiration for robotic grasping.

Figure 1.

Examples of object manipulation in birds and insects. Manipulation of: (a) grass strips by a southern-masked weaver bird (Ploceus velatus); (b) cotton strings by a zebra finch (Taeniopygia guttata); (c) mud pellets by a barn swallow (Hirundo rustica); (d) a worm by a group of wood ants (Formica rufa); (e) a flower by a bumblebee (Bombus terrestris audax); and (f) a leaf to lay eggs in by a leaf-rolling weevil (Cycnotrachelus roelofsi). Photographs by (a) B Dupont, (b,c) S Sugasawa, (d) P Dickson, (e) DJ Pritchard and (f) Y Higuchi. (Online version in colour.)

2. Seeing objects before contact

Object manipulation starts with gathering sensory information about extrinsic (e.g. distance, angle) and intrinsic (e.g. size, shape) properties of the object. How primates, birds and insects take in such information is, however, rather different. Before coming into contact with the object, primates rely primarily on visual information to generate motor commands for their forearms and hands to reach the object (figure 2a, left). This process is flexibly controlled so that if the environmental conditions change (e.g. the object moves), the animal can adjust the hand trajectory accordingly, which plays a crucial role in tool use [19]. While primates have an entire view of the object and their hands interacting with the object, viewing both from a distance, birds and insects have both their eyes and main manipulators attached to their head (figure 2a, middle and right). This means that, ironically, primates have a ‘bird's eye view' of the manipulation process (figure 2b, left), while birds do not (figure 2b, middle). Such sensory systems that are unique to each animal's perceptual and motor characteristics are referred to as ‘umwelt' [20]. Taking account of variation in these sensory characteristics would help us understand differences we see in behaviour of different animals [20]. For birds and insects, moving their manipulator also moves the eyes, and as a result the information they can access. While this arrangement might not allow the all-encompassing view of manipulation experienced by the primates, it affords birds and insects direct sensory information about the position of their manipulator relative to the object. By perceiving changes in the relative size and angles of the object as they move their manipulator towards the object, birds and insects can use simple heuristics (e.g. keep an object's centre of mass in lower centre of visual field) to grasp the object in an optimal fashion. This means that, as long as they keep their eyes open, they can control their manipulator motions flexibly. Large-billed crows, for example, kept their eyes open during reaching, quickly adjusting the pecking trajectory even when they were fitted with an artificial bill extension to grasp food successfully [21]. Also, while pigeons show a relatively fixed response in grasping tasks as they squint their eyes during reaching [22], they can adjust their motions to the size of the target object, as long as they are given a chance to see the object before reaching [23].

Figure 2.

Description of physical and sensory components of manipulation with and without hands. (a) A human, a bird and an ant engaging in a similar task (i.e. picking up food items). White arrows show the direction of force applied by different parts of manipulators, specifically, the person's index finger and thumb, the hen's bill and the ant's mandibles. (b) Even in similar manipulative tasks, the ways the person, the bird and the ant take in and process sensory information are different. The contacts between the manipulators and the object are marked with white circles, and the approximate range of visual fields is marked with a lightly shaded area. While the person has an entire view of the hands, the eyes and manipulators of the bird and the ant are attached to the head, providing direct sensory information about the position of their manipulator relative to the object. Data from manipulation in these biological systems may provide inspiration for designs (c) and necessary sensory input and operation sequences (d) of robotic grippers. All photographs used in this figure are royalty free. (Online version in colour.)

3. Sensing objects while grasping

Once a primate grasps the target object, mechanoreceptors on the hand provide rich tactile information to amass object properties like shape, substance, and texture [24,25], which, combined with proprioceptive information about the position and shaping of the hand, help them ‘work out’ how to place the manipulator for a stable grip. Within primates, the capacity for a stable grasp of objects may differ among species [26,27], potentially due to variation in their ecology [24,25]. Birds and insects have a similar haptic-sensing system to that of human hands, in the form of bills and antennae, as they both combine touch and proprioception. Birds that perform elaborate manipulations possess a highly developed somatic perception system in their bills, called bill tip organs [28–30]. In parrots, for example, these somatosensory receptors are densely distributed on the bill tip and may assist them to establish a stable grip on a tree trunk while climbing, even though they are unable to see the trunk due to the extreme curvature of their bills [30]. This way tactile information may help birds overcome the limited visual coverage of the target object, improving accuracy and efficiency of manipulation.

In fact, even with their developed eyesight, both primates and birds require tactile feedback for efficient manipulation. Not dissimilar to prosthetics users who experience a grasping problem from the lack of tactile feedback [31], injured apes lacking digits struggle to feed at the rate of intact individuals depending on the types of food [32], and chickens with clipped bills reduce their foraging rate by 80% [33], although latter response may also be explained by the birds experiencing the pain of clipping.

Similarly, insects rely heavily on touch and proprioception (figure 2b, right) to control grasp. This makes sense considering insects often handle objects that are far larger than their bodies, and so exceed their visual field. Insects usually make the first contact with food with the antennae, then follow with a series of probing movements before attempting a grasp. Insects' antennae hold a range of different tactile sensors [34]: tactile hairs detect contact, with different mechanical thresholds and directional sensitivity, while a specific chordotonal organ in antennae detects motion of the distal joint, which makes antennae active sensors. Similar sensors are found on the mandibles used for grasping and insects can be observed to ‘handle' objects before finalizing their grasp. Although it is unclear if the handling behaviour is to assess quality or to find a good gripping point, insects do respond to ergonomic properties of objects. Ants, for instance, preferentially use the ‘handle' provided by the elaisome on a seed, and when handles are artificially added to seeds the probability of seed transport increases [35]. Leaf- and grass-cutting ants position the item in the grip to improve the load balance and adjust the head angle dynamically according to the terrain on which they walk [17].

4. Neural basis of grasping control

How the brain processes sensory information to control grasping may differ between animals. Comparing the neurological architecture for grasping in primates, birds and insects would help identify how similar the pathways are between primates and birds, at which point differences become distinct, as well as how insects solve similar manipulative problems with a much smaller number of neurons. In both primates and birds, the cerebellum is involved in motor control of the processes of reaching and grasping, and cerebellar structural complexity is associated with the degree of manipulation expressed by different species [36,37]. This may mean either that the role of the cerebellum in manipulation is conserved across vertebrate groups to some extent, or the convergence in the neural underpinning of manipulation in primates and birds, despite differences in the manipulator and sensory anatomy involved. But determining exactly which would require further investigations.

In contrast, insects have drastically different neural structures from vertebrates. For manipulation by insects, there is good evidence that the insect central complex (CX) circuit is involved. For example, lesions and genetic modification of specific parts of the CX selectively affect targeting behaviour in flies crossing gaps [38], cockroaches using their antennae to anticipate obstacles [39], and praying mantis pursuing prey. All of such targeting behaviour requires the animal to estimate their precise spatial relation to a target in three dimensions and take a well-timed and well-coordinated action towards it. Also, the organization of the various modules of the CX, such as the fan-shaped body and protocerebral bridge, are more elaborate in nest-constructing social insects and insect species that perform complex independent limb movements, whereas they tend to be reduced in species that have simpler or more symmetrical patterns of movement [40]. Additionally, the dorsal lobe antennal mechanosensory and motor centre control the antennal motion, suggesting that this is the location for active sensing, and hence a useful focus for further attention for its role in grasping.

5. Action-based representation of grasping

As an animal contacts the target object with its manipulator, it kicks off a sequence of transient states of grip. Using visual and tactile feedback, the animal might regrasp and adjust the way it holds the object, to gather more information about the object, and/or to establish a more stable grip. In human hand research, each of the grips in this sequence is defined based on which parts of a hand touch the object [41]. Considering that a hand has six distinct parts including a palm and five digits, the number of possible permutations of parts that could touch the target object soars quickly, resulting in numerous entries in primate grasping repertoire. Does this diversity, however, really reflect variation in the function and the underlying control mechanisms of grasping?

Even when all digits are in contact with the object, whether all the digits touching the object in a grip play the same role depends on the task. For instance, on occasion all the digits and the palm provide force to grasp a bottle, while on others only the thumb and the index finger provide essential force to grip, and the other digits just ‘touch' the bottle. In the latter, the thumb and the index finger provide the functional grip, whereas others merely add support, or add almost no force at all. By taking account of the amount and direction of force each component applies, one can reorganize classic grip types functionally. And more importantly, describing grip using parameters like vectors of force (figure 2a) could illustrate grip in terms of the dynamic relationship between the manipulator and the object (an analogous example of tool-use definition [42]), making grasping by hands comparable with grasping by non-hand manipulators like bills and mandibles.

An obvious example of this is grasping by bird bills and pinching by hands. They both stand for the state in which an object is held between two components, a finger and the opposing thumb in a hand [41], or upper and lower beaks in bird bills, with each component exerting force towards the other. Interestingly, among primates modern humans are considered uniquely capable of strong pinching, which aids tool making and using [43], while ‘pinching' is a usual form of grasping for many birds.

This, however, does not mean that ‘pinching' is the only form of manipulation birds can do. Actions like prying or pecking, for instance, also play an important role in foraging of some birds such as crossbills and woodpeckers. Further, birds can vary their grasp by changing where on the object they hold and so how to apply the force. In fact, some birds seem to develop preferences for a specific way of holding the target object. Some male weaver birds, for instance, consistently preferred a specific side of their cheek on which they held grass to insert into a nest [44]. Similarly, individual New Caledonian crows have a preferred side against which they hold a foraging tool [45]. Such preference could streamline the control process of grasping, by always taking sensory information from the same side, and so over time, making it easy to detect deviation from the usual set of visual and tactile information. In addition, birds and insects can expand their repertoire of grips by using other body parts, increasing the number of body parts that support the target object. Parrots, for instance, are some of the few birds that have intrinsic muscles to move tongues dexterously [46]. These birds use their fleshy tongue to rotate a food item like a seed while holding it in the bill, and to stabilize it in a place while crushing the shell with its bill [46], whereby the tongue acts like a digit. Additionally, feet and legs can play a static role such as stabilizing an object, just like the thumb and the middle finger while using chopsticks (the index finger plays the dynamic role of opening/closing the chopsticks). This is indeed the case when a raptor or a praying mantis tears prey with its bill or mandibles, while holding the prey in its feet or legs.

6. Methodologies towards manipulation beyond hands

What new methods could contribute to building a broader framework to compare and integrate manipulation with and without hands? We suggest that one promising way is to make use of cutting-edge image-tracking methods, which allow describing and analysing a grasping process in terms of motions (figure 2a). The technology to record, extract and analyse behavioural footage has become cheaper and more powerful in recent years, including an upsurge of image-tracking methods to extract motions from footage (reviewed in [47]). Well-known examples include DeepLabCut [48] and DeepPoseKit [49], both of which use deep-learning algorithms to recognize the tracking target even when it changes its shape and colour with movement. More suitable for analysing how different skeletal parts move together is XMALab [50], an integrated program to collect motion data from two-dimensional X-ray videos and to reconstruct three-dimensional motions.

These image-tracking methods produce large amounts of motion data, often in the form of the target locations in each frame. Such data can be transformed into different parameters like velocity and orientation [51], parameters that can then be analysed using methods developed in movement ecology and engineering, such as hidden Markov models [52] or spline regression models [53]. These methods categorize distinct behavioural elements, by detecting systematic and recurring patterns in motions. Image tracking and clustering of behavioural elements would enable us to visualize different aspects of motions like force and speed during manipulation. In so doing, these methods would help us transfer what could only be captured as behavioural accounts into quantitative data, just like spectrograms converted bird songs into acoustic data [54]. Indeed, analytical tools that have been developed for categorizing human grasping actions might be successfully redeployed to this wider context.

7. Robotic applications of manipulation with and without hands

Grasping and manipulation have always been central to the study of robotics, but the problems of manipulation in unconstrained environments are yet to be solved [55,56]. A robot typically follows a similar series of steps in grasping as animals do: ‘sensing the object and environment' (constructing an accurate representation of the space), ‘reaching and grasping' (selecting an efficient and effective action) and ‘sensing the outcome' (assessing the success or failure of actions). Success on this task remains limited, with higher performances ranging from around 80% in rather constrained situations (e.g. only translation and rotation of the gripper needs to be determined before grasping [57]) to nearer 50% in dense 3D clutter with naturalistic objects [58]. In contrast to these low success rates of robots, animals show generally high success rates in grasping (e.g. greater than 98% in tool grasping by New Caledonian crows [59]). Learning how animals' manipulators are built and how animals move the manipulators could provide hints for designing new hardware and software for more successful robotic manipulation.

As such, a number of bioinspired approaches have been tested in robotic grasping, but perhaps surprisingly, neither insects nor birds have received much consideration as a potential model [60,61]. Some bioinspired approaches have focused on the human hand, to develop robotic systems able to operate in unstructured environments, arguing that it is important to make better use of integrated position, force, tactile and proximity sensing [62]. But if that is the case, these are equally compelling reasons to consider manipulation by mandibles as inspiration (figure 2a,b). Currently, most bioinspired work focuses on use of soft or compliant actuators [63,64], such as grippers inspired by soft fingers [8]. Similarly, inspired from unique flexible manipulators of some animals (e.g. elephant trunks, octopus arms and monkey tails), some attempts have been made to model and design robotic actuators based on these prehensile appendages [65,66]. But as the design and control of such devices for specific applications remain a major challenge, an obvious alternative is to consider animals such as birds and insects that use rigid structures to grasp (figure 2c). As birds and insects manage skilled manipulation only using their structurally simple manipulators and fewer neurons, their bills and mandibles could inspire the development of efficient robotic mandibles in a novel way (figure 2d). For instance, most primates share the basic structures of wrists (the base of the manipulator [67]), while birds are exceptionally variable in the structure of their manipulator's base, the number of cervical vertebrae [68]. This might mean that a bird's neck structure corresponds to the manipulative task (e.g. picking up a seed, tearing soft tissue or weaving a grass nest) in which the species engages, providing a starting point for designing specialist robotic bills.

8. Conclusion

Effective manipulation of objects is essential for many animals' survival and breeding. While the vast majority of manipulation research has revolved around primate hands, most animals lack hands as a manipulator. By actively integrating theories and findings from previous research to include manipulation without hands, we will gain a more general insight into how animals interact with the physical world. Such research could both refine our understanding of what is special about human hands, and inspire novel designs for grasping robots or prosthetic grippers. Together, these avenues of research into manipulation with and without hands can serve as a link between the fields of animal behaviour, animal cognition, functional morphology and biomechanics.

Data accessibility

This article has no additional data.

Authors' contributions

S.S. conceived the concept for the review. S.D.H. and B.W. contributed to the original text, with S.S. assembling the initial draft. All authors jointly edited the paper.

Competing interests

We declare we have no competing interests.

Funding

This work was supported by BBSRC Discovery Fellowship (BB/S01019X/1) to S.S.