Controlling free flight of a robotic fly using an onboard vision sensor inspired by insect ocelli

Abstract

Scaling a flying robot down to the size of a fly or bee requires advances in manufacturing, sensing and control, and will provide insights into mechanisms used by their biological counterparts. Controlled flight at this scale has previously required external cameras to provide the feedback to regulate the continuous corrective manoeuvres necessary to keep the unstable robot from tumbling. One stabilization mechanism used by flying insects may be to sense the horizon or Sun using the ocelli, a set of three light sensors distinct from the compound eyes. Here, we present an ocelli-inspired visual sensor and use it to stabilize a fly-sized robot. We propose a feedback controller that applies torque in proportion to the angular velocity of the source of light estimated by the ocelli. We demonstrate theoretically and empirically that this is sufficient to stabilize the robot's upright orientation. This constitutes the first known use of onboard sensors at this scale. Dipteran flies use halteres to provide gyroscopic velocity feedback, but it is unknown how other insects such as honeybees stabilize flight without these sensory organs. Our results, using a vehicle of similar size and dynamics to the honeybee, suggest how the ocelli could serve this role.

1. Introduction

Flying robots on the scale of and inspired by flies may provide insights into the mechanisms used by their biological counterparts. These animals' flight apparatuses have evolved for millions of years to find robust and high-performance solutions that exceed the capabilities of current robotic vehicles. Dipteran flies, for example, are superlatively agile, performing millisecond turns during pursuit [1] or landing inverted on a ceiling [2]. Moreover, these feats are performed using the resources of a relatively small nervous system, consisting of only 10⁵–10⁷ neurons processing information received from senses carried onboard. It is not well understood how they do this, from the unsteady aerodynamics of their wings interacting with the surrounding fluid to the sensorimotor transductions in their brain [3,4]. An effort to reverse-engineer their flight apparatus using a robot with similar characteristics could provide insights that would be difficult to obtain using other methods such as fluid mechanics models or experimentally probing animal behaviour. The result will be robot systems that will eventually rival the extraordinary capabilities of insects.

Creating a small flying autonomous vehicle the size of a fly such as that shown in figure 1 is a difficult undertaking. As vehicle size diminishes, many conventional approaches to lift, propulsion, sensing and control become impractical because of the physics of scaling. For example, propulsion based on rotating motors is inefficient, because heat dissipation per unit mass in the magnetic coils of a rotary electric motor increases as l⁻² [5], where l is some characteristic length of the vehicle such as wingspan. Combined with increased friction losses owing to an increased surface-to-volume ratio, exacerbated by the need for significant gearing, this results in very low power densities in small electromagnetic motors [6]. In addition, the lift-to-drag ratio for fixed aerofoils decreases at small scales because of the greater effect of viscous forces relative to lift-generating inertial forces at low Reynolds numbers [7].

Figure 1.

A 106 mg robot the size of a fly uses a light sensor to stabilize flight, the first demonstration of onboard sensing in a flying robot at this scale. (inset) The visual sensor, a pyramidal structure mounted at the top of the vehicle, measures light using four phototransistors and is inspired by the ocelli of insects (scale bar, 5 mm). (main) Frames taken at 60 ms intervals from a video of a stabilized flight in which the only feedback came from the onboard vision sensor. The sensor estimates pitch and roll rates by measuring changes in light intensity arriving from a light source mounted 1 m above (not shown). This is used in a feedback loop actuating the pair of flapping wings to perform continuous corrective manoeuvres to stabilize the upright orientation of the vehicle, which would otherwise quickly tumble (see §5). A wire tether transmits control commands and receives sensor feedback, acting as a small disturbance that does not augment stability. (Online version in colour.)

The challenges imposed by the small scale of the vehicle shown in figure 1 extend to sensing and control. The rate of rotational acceleration increases with decreasing size, scaling as l⁻¹ [8], requiring a feedback sensor with higher bandwidth [9]. Silicon microfabricated gyroscopes and accelerometers have recently become small enough to be carried on such vehicles, primarily because of pressure from the consumer electronics industry. However, their signal fidelity may be significantly disrupted by the vibratory environment on vehicles with flapping wings [10], particularly as the flapping frequency increases at smaller scales. At a higher level of control, perceiving attitude and position relative to obstacles or targets will be essential to achieving autonomy. The global positioning system used by larger aircraft becomes impractical at small scale because of insufficient resolution (1–10 m) and signal disruption in cluttered and indoor environments [11]. Emissive sensors such as a laser rangefinder are currently too heavy—the lightest commercially available scanning rangefinder has a mass of 160 g (Hokuyu URG-04LX-UG01), three orders of magnitude heavier than the vehicle in figure 1—and require too much power [11]. Vision provides a promising solution: cameras can be made small and low power [12], but, because of the restricted onboard processing power available, computational requirements must be minimized. Although the rate of translational acceleration remains constant as scale decreases [8], obstacles may be nearer, requiring high frame rates and faster processing. Despite the difficulties imposed by small scale, demonstrations targeted at such vehicles, to name a few, include navigating confined spaces [13–15], a high-frame-rate (300 Hz) omnidirectional camera [16], obstacle avoidance using monocular vision [11,17] or stereo [18], altitude regulation [19,20], hovering [21] and an initial implementation on a robotic fly [12].

The first fly-sized robot to lift its own weight took inspiration from flying insects [6,22]. For the sake of mechanical simplicity, this vehicle has two hoverfly-inspired wings rather than four, driven by piezoelectric actuators performing a muscle-like reciprocating action [23]. Piezoelectric actuators were chosen because they scale down more favourably than electromagnetic motors [5], and the flapping motion generates unsteady aerodynamics that enhance lift [22]. This prototype is now being used to better understand and optimize the relevant fluid mechanics [24,25].

When first flown without guide wires, however, the robotic fly tumbled [26]. This suggested that, to remain aloft, it requires constant corrective feedback as is required in unstable fighter jets [27]. Simulations have suggested that insects have a similar instability [28–32]. Because of the fast rotational dynamics [8], control theory dictates that any stabilizing feedback controller must have a short time delay [9,32]. Controlled flight demonstrations have so far relied on an array of external cameras to precisely triangulate the position and orientation of the vehicle equipped with reflective markers [33], but this approach cannot extend beyond specific laboratory conditions. Stability has been achieved with aerofoils such as air dampers [34] or a tail [18], but this makes the vehicle susceptible to wind disturbances and sacrifices manoeuvrability.

Here, we address the challenge of achieving stability of a robotic fly by considering an approach inspired by flying insects. Most flying insects have three light sensors, distinct from the compound eyes, that point roughly upwards and sense light from the sky and/or the Sun [35]. These three sensors, known as the ocelli, are relatively defocused and carry many neurons that sample the same light information [36,37]. With this optical apparatus, they are much more sensitive to light than the compound eyes [36], allowing them to quickly sense changes in the light levels to aid in rapid self-righting manoeuvres. However, the precise role they play in flight stability is not well understood [3,38,39]. While dipteran flies are thought to stabilize flight in part by using mechanosensory feedback from their elaborate gyroscopic halteres [40,41], other insect orders have no such apparatus. Hawkmoths use large vibrating antennae to sense rotations using a similar mechanism [42,43]. However, the process by which other species stabilize their flight, such as honeybees, remains unknown [20]. It is possible that these animals' flight stabilizer relies heavily on the ocelli.

From the perspective of autonomous flight control, an ocelli-inspired light sensor is nearly the simplest possible visual sensor, minimizing component mass and computational requirements. A number of previous studies have considered ocelli-inspired sensors on flying robots, insect-sized [44–47] or larger [48,49]. In [50], it was shown that an adaptive classifier could be used to estimate the orientation of the horizon from omnidirectional camera images. In [44–48,50], the absolute direction of the light source or horizon relative to the vehicle was estimated. But whereas aligning to the absolute direction of a light source or horizon may be a valid approach for larger aircraft that fly above obstacles so there is a relatively clear view to the horizon, smaller vehicles may fly near buildings, under foliage or indoors. In these conditions, the horizon may be obstructed. This causes the direction of light sources to vary significantly [45]. A control law that aligned the vehicle with a light source would most likely yield a tilted vehicle under these conditions, leading to significant lateral acceleration and dynamic instability.

In this work, we propose an alternative approach in which a feedback controller applies torque in proportion to angular velocity of the motion of the source of light. This has two benefits. First, it avoids the need for the light source to be directly above for the sensor to produce a useful result. Second, we show that an angular velocity estimate is all that is needed to stabilize the upright orientation of our flapping-wing robotic fly and many flying insects. The first was confirmed by previous work, inspired by observations of derivative-like responses in insect ocelli [36,37], that showed that ocelli simulated in a virtual environment can estimate angular velocity about the pitch and roll axes, regardless of initial orientation [49]. The results also suggested that a linear ocelli response cannot estimate other vehicle motion parameters such as absolute attitude. A motor controller was described that computed a time integral of the ocelli angular velocity estimate. Although this did not require the light to arrive from a known direction, an estimate computed in this manner would slowly drift because of accumulated sensor noise. Here, we build on that work to suggest an alternative approach in which the angular velocity estimate is instead used directly in a feedback controller. By applying torque in proportion to angular velocity only, it is possible to harness this vehicle's flapping-wing dynamics to achieve a stable upright attitude that does not drift and that does not require absolute estimate of attitude.

In §2, we describe the design and fabrication of the ocelli-inspired sensor and flapping-wing robotic vehicle. In §3, we give a model of the flapping-wing vehicle that shows how its dynamics can be stabilized by angular velocity feedback, also known as ‘rate damping’, for both the pitch and roll axes. This is possible, because, like many flying insects [32,51], the centre of mass (CM) of the vehicle shown in figure 1 hangs below the wings [33]. This gives rise to unstable pendulum-like dynamics that can be stabilized by damping. In §4, we build on previous analyses of four-sensor ocelli designs to show how to estimate angular velocity using an arbitrary number of light sensors. This could allow the number to reduce to three, as in most flying insects, to save weight. In §5, we demonstrate this feedback law in operation by stabilizing a fly-sized robot using only feedback from the ocelli-inspired light sensor. We conclude with a discussion about how this work suggests how animals without gyroscopic feedback, such as the honeybee, could use the fast, derivative-like reflex mediated by the ocelli [3,36,37,52] to perform an equivalent role to stabilize their flight dynamics.

2. Sensor and robot fly

2.1. Ocelli sensor design and fabrication

The design of the ocelli sensor (figure 2 and shown attached to the vehicle in figure 1) draws direct inspiration from insects' ocelli, as in previously described prototypes [48,49,53,54]. Each sensor is inclined roughly 30° above the horizon (as in flies [55]) and captures defocused light from an angular field spanning approximately 180° that is nearly circularly symmetric.

Figure 2.

The ocelli design consists of four phototransistors soldered to a custom-built circuit board that is folded into a pyramid shape. (a) The unfolded circuit board before surface-mount components have been added. The transistors' emitters are connected together (centremost trace) and tied to ground. (b) The larger translucent components are the phototransistors and the smaller components are 0201-sized resistors. Lines denote the orientation directions of the sensors. The device weighs 25 mg and is shown beside a US 1 cent coin for scale. (Online version in colour.)

Our ocelli sensor is custom-fabricated to obtain small mass (25 mg) and small footprint (4 × 4 × 3.3 mm) compatible with the robotic fly's payload capacity. The basic structure of the device is formed by a lightweight folded circuit board. Traces were fabricated by laser-ablating the copper layer of copper-clad polyimide. Subsequently, the outline of the board was cut by laser, and a unidirectional carbon fibre layer was adhered to the back to increase structural stiffness. Surface-mount components were soldered on by hand, and the pyramid structure was then hand-folded into shape. Each of the four light detectors consists of a phototransistor (KDT00030 from Fairchild semiconductor) in a common-emitter configuration in series with a 27 kΩ surface-mount resistor. The phototransistor has an infrared cut-off filter, reducing its sensitivity to the bright infrared lights emitted by the motion capture system used to measure flight trajectories. The voltage reading is taken from the collector of each transistor and rises with increasing luminance.

2.2. Robotic fly mechanical characteristics

The flying vehicle used in this work (figure 1) is actuated by a pair of independently moving wings. By altering signals to the piezoactuators driving the wings, they can produce sufficient lift to take-off, as well as produce ‘pitch’ and ‘roll’ torques independently. Greater detail about its mechanical design is given in [33,56].

We define a right-handed coordinate system for the body in which, with the wings extending laterally along the y-axis and the body axis hanging downwards in the negative z-direction, the x-axis (roll) points forward, the y-axis (pitch) points to the left and the z-axis (yaw) points upwards (figure 3). Roll torque is induced by varying the relative stroke amplitudes of the left versus right wing. Pitch torque is induced by moving the ‘mean stroke angle’—the time-averaged angle of the forward–backward motion of the wings—in front (+x) or behind (−x) the CM [56]. Yaw torque can also, in principle, be modulated [56], but we do not use this capability in this study.

Figure 3.

Model for the lateral planar dynamics of the robotic fly. (a) The vehicle state is given by three quantities: the attitude angle θ, the angular velocity and the lateral velocity v (given in body-attached coordinates). A lift force f_l generated by the flapping wings acts at a distance r_w away from the CM and along the body z-direction, an aerodynamic drag force f_d acts at a distance r_w from the CM, and the gravitational force mg acts at the CM. The right-handed axis convention for world coordinates is shown in the inset, with x pointing into the paper; lowercase letters denote body-frame coordinates and capital letters denote world-frame coordinates. A feedback controller applies a control torque τ_c by altering baseline wing kinematics. (b) Wind tunnel tests of a flapping-wing robotic fly indicate that aerodynamic drag is roughly proportional to wind speed for wind from the front (+x-direction, solid line) and from the side (+y-direction, dashed) for typical wing kinematics. Force measurements (mean ± s.d.) were taken using a torque sensor (Nano 17, ATI Industrial Automation, Apex, NC, USA) on a 30 cm cantilever arm. The slope of the linear fit for both cases, b_w, was modelled as equivalent for both directions and is shown as a thick line and given in table 1. (Reproduced with permission from [34].) (Online version in colour.)

In our current set-up, the robot is given power and controlled through a lightweight compliant tether wire. Power and control commands are transmitted over four thin and flexible 51-gauge (0.022 mm diameter) copper wires, each less than one-quarter the diameter of a human hair. When present, the ocelli sensor requires six additional wires to return sensor feedback: power, ground and four voltage levels from the phototransistors.

Although it is difficult to quantify the effect of the wire tether because of its variable conformation, we attempted to estimate its effects in a number of ways. We remark that by itself it cannot stabilize the vehicle: no flight in hundreds of attempts has been observed to stay upright without active feedback. Considered as a mass, the wires weigh approximately 1 mg for a 5 cm length of four (approximately the altitudes reported here). This is a small component of the 100 mg vehicle mass. Modelled as a point mass at the end of a moment arm extending laterally in the x-direction for a distance of 1 cm (the largest lateral displacement of the tether observed in this work), this would impart a fixed torque of approximately 0.1 μNm. This is smaller than the approximately 1 μNm corrective torques that can be produced by the wings [56]. This is also smaller than typical trim, which are calibrated during initial flights so that the vehicle flies vertically at take-off. As a source of aerodynamic drag, the flapping wings have a larger area (approx. 0.75 cm²) and move nearly an order of magnitude faster (≈10 m s⁻¹ at the tip) than the 5 cm wire relative to the downwash (0.04 cm², ≈2 m s⁻¹ on average at the wings, measured by particle image velocimetry), suggesting the tether has a negligible effect. Considered as a spring, we remark that the tether exhibits sufficient stiffness when the vehicle is on the ground that it sometimes caused the vehicle to tumble before becoming airborne. Data from such flights are not reported here. Once in the air, however, experiments using a precise torque sensor indicate that a four-wire tether acts as a gentle spring with a stiffness of approximately 0.25 µNm per 45° (electronic supplementary material, figure S1). When incorporated into the model as a spring, it has a small effect on its dynamics, which we consider in greater detail below. Lastly, feedback from the wings can consistently overcome any effects of the tether, because in hovering tests [33] the vehicle was able to move laterally large distances (20 cm) relative to its altitude (10 cm) without crashing, despite large conformational changes in the tether.

Difficulty of vehicle fabrication and limited lifetime constrained the number of flights to just a small number in this work. Each vehicle takes days of full-time labour to fabricate [6], and because of manufacturing or human error, some do not have the tolerances necessary to attain stable free-flight. Mechanical failure of the flexure joints in the wings limits flight times to minutes, after which the vehicle must be discarded. Although this is expected to improve through future technological developments, the results reported here are therefore concerned primarily with demonstrating feasibility.

3. Attitude stabilization using velocity feedback

We show that knowledge of absolute vehicle attitude is not required to attain stability. Instead, only angular velocity feedback or ‘rate damping’ is needed. More formally, the torque controller

3.1

where ω is the angular velocity, is sufficient to stabilize the fly in the upright orientation.

This result holds under the following assumptions:

• (1) Vehicle motions depend only on stroke-averaged forces, that is, forces and torques averaged over the time period of each wing stroke.

• (2) Aerodynamic drag on the wings is proportional to airspeed in both the forward (x) and lateral (y) directions, with an equal proportionality constant for both directions.

• (3) The vehicle is symmetric about its x–z plane.

3.1. Rigid body dynamics

We use the following quantities, all expressed in body-frame coordinates: is the velocity of the CM; is its angular velocity about the CM; is a torque applied to the body; are external forces acting on the CM; m is the mass; and is the matrix of the moment of inertia.

We use two different representations for the attitude: rotation matrices and Euler angles. Using Euler angles, the attitude is represented by an array of three angles . The attitude is obtained by first rotating by an angle θ₃ (yaw) around the body z-axis, then by θ₂ (pitch) around the new body y-axis, and then by θ₁ (roll) around the new body x-axis. This representation has singularities at extreme attitudes but is convenient to prove local stability around a certain attitude.

Using this parametrization, the dynamics of the rigid body in body-attached coordinates are

3.2

3.3

3.4

The quantity W(θ) is a matrix that relates the angular velocity ω to the rate of change in Euler angles. Rewriting equation (3.2) in terms of coordinates gives

3.5

We also represent the attitude as a rotation matrix . For any vector given in world coordinates, , where is the same vector expressed in body coordinates (figure 3). Using this representation, equation (3.2) is

3.6

where is the skew-symmetric matrix that represents the cross-product of a vector with ω, so that .

3.2. Analysis of the planar model

Assuming that the z-torque is negligible, we can reduce the analysis of the six degrees-of-freedom (d.f.) system (equation (3.4)) to two independent planar systems. We first provide the planar analysis for the sake of simplicity. The avid reader can proceed to the subsequent section for the proof for full rigid body motion.

Consider a state defined by a scalar pitch angle θ, a scalar angular velocity , and scalar torque τ (figure 3). We show that the controller,

3.7

that applies a torque proportional to the angular velocity stabilizes the fly in the upright orientation.

A key element of the model is how the aerodynamic drag on the wing acts on the rotational dynamics of the robotic fly. As the rotational dynamics are slow relative to the frequency of flapping at this scale [20], our analysis considers only stroke-averaged forces. A test of this vehicle flapping in a wind tunnel indicated that the stroke-averaged drag force on the wings is nearly linear with the incident airspeed for typical wing kinematics (figure 3). This is the case for wind in both the x- and y-directions. Accordingly, our model for aerodynamic drag in both cases is

where v_w is the lateral velocity of the point on the airframe at the midpoint between the two wings. If the vehicle is rotating at angular velocity ω, then the velocity of the wings, when linearized around θ = 0, is , where r_w is the distance from the midpoint of the wings to the CM. Similarly, the force arising from aerodynamic drag is and the torque about the CM owing to this force is . With lift force owing to the flapping wings f_l approximately balancing out the weight mg, the lateral force owing to the inclined gravity vector relative to the body frame is equal to for small θ. A simplified model of the wire tether found using a sensitive torque sensor suggests it could be incorporated into the model as a spring with constant k_s (electronic supplementary material, figure S1), according to τ_s = −k_sθ.

Next, we compute the dynamics under the influence of the torque controller in equation (3.7). In the planar motion case, we can neglect second-order cross-product terms in equation (3.4) and equate forces and torques to velocities according to and . The linearized equations of motion about zero pitch angle θ can be written as a state-space dynamical system with the state vector expressed in body coordinates, where

3.8

If desired, the effect of the wire tether can be incorporated by setting A₂₁ = k_s/J, but we neglect its effects in the following analysis.

The dynamics are asymptotically stable if the eigenvalues of A have negative real part. The Routh–Hurwitz criterion can be used to determine the stability of this system [9]. This allows us to determine the sign of the eigenvalues by looking at the characteristic equation , which gives a polynomial of the form . The dynamics are stable if and only if all a_k > 0 and a₂a₁ > a₃a₀. If k_d > 0, then all of the a_k > 0, so the stability criterion reduces to a quadratic polynomial of the form , which has two solution domains. The negative solution for k_d is not stable, because some a_k are negative. The positive solution domain is

where the bound is given by

3.9

Therefore, we have found a lower bound on the gain for the controller in equation (3.7) that is necessary to achieve asymptotic stability at .

The rotational dynamics of the robotic fly given here can be compared with a pendulum, but are qualitatively different because of how lateral and rotary motion are coupled. In the course of flying, if the body is inclined, then the thrust vector takes on a lateral component, accelerating the vehicle laterally. As lateral velocity increases, drag from the wings increases. The lateral drag acts at a location above the CM to exert a torque that acts to right the vehicle (drag on the body is negligible). In our design, this torque is large enough that it causes the vehicle to swing with a larger amplitude in the opposite direction, leading to a growing oscillation that results in tumbling. The velocity-dependent action of the controller in equation (3.7) suppresses this through a damping action [32].

To compute the necessary minimum gain, we estimated parameters of the robotic fly (table 1). Mass was measured using a precision scale, and the wing drag factor was measured in a wind tunnel (figure 3). Moments of inertia were estimated using a detailed model in computer-aided design software. The quantity r_w was estimated by measuring the distance from the leading edge of the wings to the approximate point at which the airframe of the robotic fly balanced on a sharp edge using a ruler under a microscope. Evaluating equation (3.9) using these parameters gives for both xz- and yz-dynamics. Experimental results (described later) confirm that this value is near the limiting value between stability and instability, validating our mathematical results and modelling assumptions.

Table 1.

Estimated parameters for the robotic fly shown in figure 1.

quantity	symbol	quantity	units
mass	m	81 × 10−6	kg
mass of ocelli		25 × 10−6	kg
moment of inertia (x-axis)	J1	1.42 × 10−9	kg m2
moment of inertia (y-axis)	J2	1.34 × 10−9	kg m2
moment of inertia (z-axis)	J3	0.45 × 10−9	kg m2
moment of inertia added by ocelli (x- and y-axis)		1.8 × 10−9	kg m2
wing drag factor	bw	2.0 × 10−4	Ns m−1
distance, CM to wings	rw	9 × 10−3	m

3.3. Analysis of the full six degrees-of-freedom rigid body model

3.3.1. Forces and torques

We assume that aerodynamic drag, in addition to being proportional to velocity in the x- and y-directions (figure 3 and [34]), is also proportional to velocity in the z-direction, although this has not yet been tested. In vector form, the drag force is thus where is defined as the velocity of the point midway between the two wings in world coordinates. If the body is rotating with angular velocity ω, then the velocity at that point in world coordinates is V_w = Rv + (Rω) × (Rr_w), where is the point where the two wings attach to the airframe and is approximated as being directly above the CM. Using that (Rω) × (Rr_w) = R(ω × r_w), and that R^TR = I, where I is the identity matrix, we have that

Gravity applies a force directly downwards through the CM and is given by in body coordinates. The lift force from the wings acts with a magnitude roughly equal to the force of gravity, f_l = [0, 0, mg]^T and in line with the CM. Thus, we have that the total external force applied to the vehicle, expressed in body coordinates, is

The torque applied by drag on the wings is and the control torque is τ_c = −k_dω (equation (3.1)). Thus, the entire torque is

3.3.2. Linearized analysis around hover

We use a linearized analysis to assess stability of the hover configurations (ω = 0; v = 0; θ₁ = 0, θ₂ = 0, and any value for θ₃). We use the local coordinates and derive the linearized dynamics valid around the point .

Consider the rigid body dynamics described by equations (3.2)–(3.4). Taking the derivative, the second-order cross-product terms ( and ) disappear. To fill out the matrix A, we compute the linearized dynamics and obtain that

Similarly, we obtain , and . Because our vehicle is symmetrical, its moment of inertia is well approximated by Assembling the matrix A, we obtain

3.10

We can verify that for any (equation (3.9)) A has all eigenvalues with negative real parts, except for a zero eigenvalue corresponding to free state θ₃, the heading angle of the vehicle, whose value has no effect on the dynamics. Because it is sufficient to show that the linearized dynamics are stable to prove that the full nonlinear system is stable in a neighbourhood of that equilibrium [9], we have proved that the angular velocity feedback control law in equation (3.1) stabilizes the robotic fly in the upright hovering position, provided that , with given by the lower bound in equation (3.9).

An inspection of the rows of A shows that the two planar systems and have no coupling, so that each planar system can be analysed independently. This allows for the analysis considered above. We must emphasize that this only applies for a linearization around ω₃ = 0. However, if manufacturing asymmetry generates a small but constant disturbance torque around the z-axis (this is commonly the case), then the flapping wings generate a counter-torque roughly in proportion to the angular velocity ω₃ [57], so that ω₃ does not grow without bound. Therefore, the fly will stabilize in the upright orientation, and rotation around the vertical axis will remain small.

A similar analysis suggests that stability could also be achieved through a mechanical design change that reduces r_w, the distance from the wings to the CM, assuming the wings themselves provide a small amount of rotational damping. For example, if the wings impart a torque τ = −cω with , then passive stability is achieved with 0 < r_w < 0.7 mm.

4. Estimating angular velocity using the ocelli

Here, we show how to use the ocelli output to produce an estimate of the angular velocity for use in a feedback controller (equation (3.1)) to stabilize the robotic fly. We call each photoreceptor an ‘ocellus’ to highlight the biological inspiration. We will prove the following result: there exists a matrix L such that the least-squares estimate of the angular velocity is , where is the vector of the time derivatives of each ocellus signal. The result holds under the following assumptions:

• (1) The sensitivity of each ocellus is circularly symmetric. This is a reasonable approximation for the phototransistors in our sensor prototype and fly ocelli [55].

• (2) There is only one static circularly symmetric luminance source in the environment. This is typically the case outdoors: the sun is a localized luminance source; a cloudy sky is a very diffuse luminance source taking on a hemisphere.

• (3) This light source is far away. With this assumption, translational motion of the vehicle does not induce a significant change in the direction of the light source relative to the vehicle.

• (4) There are at least three ocelli, arranged in a non-singular configuration.

Previous work in the literature has considered the case in which the ocelli are configured in opposing pairs [45,49]. Here, we give a geometrical treatment that provides for an arbitrary number of ocelli in arbitrary directions, permitting either greater feedback precision or a reduced sensor mass as necessary.

4.1. Ocelli sensor model

Let the sensor be composed by n defocused photoreceptors. The output of the sensor is an array in which each component is the voltage signal from a single ocellus, all of which have identical response properties. This voltage is proportional to the luminance in the environment averaged over the visual sphere in the directions to which the ocellus is sensitive.

Let be the environment ‘map’, so that m(s) is the luminance in direction on the unit sphere. We assume this is unchanging according to assumption 3.

Let be the principal direction of the ith ocellus in body coordinates. If the current sensor attitude is , then the ocellus is pointing in the direction Using the assumptions outlined above, it can be shown¹ that the output of the ith ocellus at time t is a function only of the angle between the centre of the light source d,

4.1

This assumption has been verified, in practice, for many lighting conditions when using radially symmetric and defocused light sensors such as those used in this work (assumption 2) [45].

4.2. The derivative of the ocellus signal is a linear function of angular velocity

Given the sensor model in equation (4.1), we can obtain an estimate of the angular velocity ω. We start by taking the derivative with respect to time of the sensor output. We assume the visual scene is static according to assumption 3 so that . We obtain

The last equality comes from the fact that is linear in R_t so [58].

We can linearize around the reference point R = I to obtain

4.2

This expression is a linear function of ω_t, so we can find a matrix such that . Using the standard vector triple product and dot product identities, we obtain and rewrite equation (4.2) as

4.3

Therefore, we have concluded that the derivative of the ocellus signal is related linearly to the angular velocity.

4.3. Least-squares estimation

We have obtained the expression in equation (4.3) for the derivative of one ocellus signal. If we write it together for all ocelli, then we obtain that the derivative of the vector depends linearly on the angular velocity ω_t through an n × 3 matrix M,

4.4

The least-squares estimate of ω_t, which assumes equal noise on all measurements, can be written as

4.5

This estimate has a number of restrictions. First, the component of ω_t parallel to d cannot be estimated because it is in the null space of M. This is because each row of M is the result of a cross-product with d (scaled by κ′), so each is orthogonal to d. Together, these rows constitute a plane that is the kernel of this matrix, and the pseudo-inverse provides only the projection of ω_t onto this plane. This subspace is further restricted to a line if d and all sⁱ are coplanar, because, in that case, all rows of M are linear multiples of each other. Second, the row M_i is zero whenever d = ±sⁱ, because, at these points, d × sⁱ is zero. It is also zero when κ′ is zero for any ocellus, such as when the light source is entirely outside of its field of view, so that angle changes have no effect on its response. Third, the magnitude of the estimate varies linearly with κ′ and thus with the brightness of the light source function κ.

This formulation allows for an arbitrary arrangement of an arbitrary number of ocelli. As few as two can estimate the two essential components of to provide stability if the luminance function κ is known. If it is not, then redundant sensors can be used to estimate κ [47].

4.4. Specifics of the four-ocellus design

In our design, the directions of the four ocelli are closely approximated by and . This arrangement has the following characteristics:

• (1) The ocelli are arranged in pairs that are redundant, so that the overall illumination level κ could be estimated and divided out, as in [47].

• (2) They are tilted in a slight upwards direction, so that a light source above the vehicle will be detected by all of them in the course of normal manoeuvring so that . A tilt angle of approximately the same value, 40°, was found to be optimal for estimating angular rate in [49].

• (3) None is tilted so far upwards that it will be pointed directly at the light source in the course of normal manoeuvring so that .

• (4) The pairs are arranged orthogonally to provide orthogonal components of ω, simplifying control design [44,45].

In the case of the light source directly overhead the M matrix (4.4) is given by

In this case, the quantity ω₃ (the z-component of ω) is not observable, because d is directly above, but for hovering it is sufficient to leave it uncontrolled, so long as it remains small. For the other two components, the matrix pseudo-inverse has the form

4.6

where is a constant. A new output defined by subtracting opposing pairs of ocelli according to [45] recovers a simplified relation similar to that given elsewhere for a pair of ocelli [49],

4.7

If the light is not directly overhead, but at some unknown angle, then the quantity k may be different for that orientation. However, by combining equations (4.7) and (3.1), it can be seen that as long as it can be ensured, by making k_d sufficiently large, that , where k = k₀ when the light source is directly overhead, then the control law given by (3.1) is asymptotically stable.

5. Flight tests

5.1. Flight arena

We performed flights tests in a motion capture arena with an array of calibrated cameras (T040-series; Vicon, Oxford, UK). Each camera emits bright infrared that is reflected from a number of retroreflective markers mounted on the vehicle, so that its position and orientation can be reconstructed in real-time for later analysis. Rotation about yaw (z) was left uncontrolled in these experiments. In all flights, computations to generate signals for the piezoelectric actuators to drive wing motion, as well as to map desired control torques to these signals, were performed on an XPC Target, a desktop computer running a real-time operating system (MathWorks, Natick, MA, USA). Analogue voltage outputs from this computer were amplified by high-voltage amplifiers and transmitted to the robotic fly through the wire tether. The compliant wire tether has a small effect on vehicle dynamics, as detailed in §2.2.

In flights controlled by feedback from motion capture, position and orientation estimates were sent by serial cable to the control computer. In these tests, the estimate was calculated by smoothing the motion capture estimates with a third-order Butterworth filter with a 40 Hz cut-off frequency, taking the derivative, and multiplying by the matrix W (equation (3.5)).

For flights in which feedback from the ocelli was used, illumination was provided by a 150 W halogen bulb at 66% brightness mounted 1.5 m above the vehicle (Dolan-Jenner MI-150 fibre-optic illuminator; Edmund Optics, Barrington, NJ, USA). The control computer measured voltages from the four phototransistors using an analogue-to-digital data acquisition board (National Instruments). The infrared flashes from the motion capture cameras, at 500 Hz with a one-eighth duty cycle, were so bright that they saturated the phototransistors, regardless of orientation because of reflections from the ground surface. Accordingly, they did not provide any orientation information. After this initial transient, the voltage decayed to an attitude-dependent steady-state value (figure 4). A digital filter recovered these steady-state values by filtering with a third-order Butterworth filter with a 20 Hz cut-off frequency to smooth out the sudden flashes, as well as the 60 Hz AC ripple from the light source (figure 4). To calculate the estimate of k in equation (4.7), we rotated the robot by 45° in either direction and measured the resulting quantity y₄ − y₂ to compute the slope.

Figure 4.

Calibration of the ocelli-inspired visual sensor. (a) Voltage readings y₂ (black) and y₄ (grey) from an opposing pair of phototransistors, before filtering (the effects of the one-eighth duty cycle, 500 Hz infrared flashes of the motion capture cameras can be seen as spikes followed by a short transient decay) and after (smooth trace showing trend), taken during a calibration in which the robotic fly was held by tweezers as it was rotated by hand in the motion capture arena. (b) To measure the approximate acceptance profile of the phototransistor, it was rotated under the fibre-optic light source, approximately a point source. The response spans a range of approximately 160° (black) and resembles a Gaussian function with σ = 61° (thick grey line, fit performed by fminsearch). (c) y₂ (black), y₄ (grey) and y_b = y₄ − y₂ (dashed), plotted against measurements from the motion capture system. The slope of this line at zero angle corresponds to k in equation (4.7). A third-order polynomial fit of angle θ₂ to y_b using the Matlab command polyfit is shown (thick grey line). (d) This polynomial was used to estimate θ₂ from the ocelli responses at θ₁ = 0 (red) compared with the estimate from motion capture (black) and shows a close correspondence for the range of ±60°. If the ocelli are inclined about the other axis at θ₁ ≈ 30°, the estimate is not significantly perturbed (dashed grey line). (Online version in colour.)

5.2. Results

Experiments provide empirical evidence that the control law given by equation (3.1) gives stability. In flights without the ocelli attached in which the estimate was provided by motion capture using a gain of k_d = 2 × 10⁻⁷, well into the region of stability, the robotic fly remained upright during the course of the flight before drifting out of the tracking volume of the motion capture arena. In these flights, only the rate damping feedback law was being tested: neither altitude nor lateral position was under feedback control so small asymmetries in flapping kinematics generated non-zero lateral thrust.

To further test the validity of the model given by equation (3.8), we observed the natural dynamics of the robotic fly in flight with a gain of k_d = 1.0 × 10⁻⁷, just at the threshold of stability. In this flight, it exhibited sinusoidal oscillations similar to those of a simulation of the model in equation (3.8) (figure 5). When tested with gains of 0.75 × 10⁻⁷ or lower, the vehicle did not remain upright. This shows that the model and parameters given in table 1 are close to their true values and that un-modelled effects do not have a large impact on vehicle dynamics.

Figure 5.

An empirical test of the robotic fly in flight with angular velocity feedback alone from motion capture, with a gain of k_d = 1.0 × 10⁻⁷ (just at the threshold of stability), shows that it undergoes rotational oscillations. The unfiltered Euler angles θ₁ and θ₂ measured by motion capture (rotation about body x-axis and y-axis, respectively) are shown in black and grey. A linear simulation of the robotic fly dynamics (equation (3.8)) with the same gain, an initial lateral velocity of 0.3 m s⁻¹ (thick black line), and parameters from table 1 shows similar behaviour, providing empirical support of the model proposed in equation (3.8). If the wire tether is incorporated into the model as a torsional spring (§3.2), then it has a small effect on dynamics (dashed line). Other differences may be due to effects not accounted for in the model such as un-modelled aerodynamic forces.

A demonstration that the robotic fly can be stabilized exclusively using feedback from the ocelli is shown in figure 6 and in the electronic supplementary material, video S2. In this experiment, the ocelli sensor was attached to the vehicle and provided the estimate . The added mass and inertia of the ocelli increased the minimum gain to . Using a gain of k_d = 2 × 10⁻⁷ as above, the fly successfully remained upright during a 0.3 s climbing phase (approx. 40 wingstrokes) before the wiring reached its limit. The vehicle remained upright, but performed rotational oscillations about pitch and roll, the result of aggressive corrective manoeuvres. Without this feedback, the vehicle quickly tumbled (figure 6, inset). The oscillating instability in the ocelli-stabilized flight was likely to be due to the nonlinearity of torques exceeding the pre-programmed actuator limit, owing to an overestimate of near the upright orientation. In addition, if at take-off the vehicle was inclined at 30°, then this fell outside the range in which the estimate of was valid, causing tumbling. The maximum inclination during this flight was approximately 10°, suggesting that torques owing to the torsional stiffness of the wire tether (§3.2; approx. 0.16 µNm) should have a small effect relative to control torques.

Figure 6.

Take-off in which the ocelli were using a purely linear feedback law. (a) The camera was aligned with the positive y-axis and the time interval between frames is 66 ms. Without feedback, the vehicle quickly tumbles (inset) (also see electronic supplementary material, video S2). (b) Coordinates of flight trajectory estimated by motion capture: z (solid), x (dashed) and y (dash-dotted). (c) The angular velocity ω measured by motion capture (solid lines) versus estimate from ocelli (dashed); is black and is grey. Ripple owing to the flapping wings can be observed in the pitch rate estimate. At small θ₁, the estimate of roll diverges from the true value. (d) Pitch and roll torque commands from the controller (solid) are inversely proportional to angular velocity; a safety saturation block limited the control command to the wings. (e) Trajectory measured by the motion capture system rendered at every 20 ms. The vertical line is equal to the length of the vehicle and denotes the direction of its long axis. Grey lines in the background show projections of the trajectory onto the xy, xz and yz planes, and the blue line is a projection of the vehicle x-axis onto the xy plane to show the vehicle's heading. (Online version in colour.)

To mitigate the effect of variable gain and consequent actuator saturation, and to extend the range of permissible inclination angles, we performed third-order least-squares fit of θ₂ measured by motion capture to y_b across a range of ±60° using the Matlab command polyfit, using this fit for both axes. During flight, evaluating this polynomial requires minimal additional computation. With this calibration, the region of operation expanded from roughly ±30° to ±60° (figure 4). A motion capture estimate is not necessary, in principle, to perform this calibration; it could easily be performed using a potentiometer, as in [47]. In flights with this calibration, the orientation was more stable and the estimate matched the motion capture estimate more closely (figures 1 and 7). Although this calibration improved performance for only a specific lighting condition, it is possible this could be extended to improve performance in more general conditions. This could be achieved by performing a least-squares fit using data taken from a range of expected lighting scenarios. Our model of the wire tether indicates it could have imparted torques as high as 0.24 µNm when the vehicle reached an 18° inclination, about half the magnitude of the control torques in this trial.

Figure 7.

Data from take-off flight shown in figure 1. In this flight, the ocelli were calibrated to have a larger operating regime. The lower gain avoids actuator saturation and rotatory oscillation. As in figure 1, (a) shows vehicle position, (b) shows a comparison of angular velocity estimates, (c) shows control torque commands, and (d) is a three-dimensional plot of the vehicle trajectory. (Online version in colour.)

To further demonstrate the utility of the proposed control law, we performed an additional flight in which this calibrated ocelli feedback was combined with the aerodynamic ‘ground effect’ to maintain altitude. The ‘ground effect’ is an increase in aerodynamic lift that occurs in close proximity to a horizontal surface [14]. In this flight, both upright orientation and altitude were maintained for a prolonged period of nearly a half of a second, or 50 wing strokes (figure 8), limited only by tether length. This result suggests that, if not for the tether length limit, ocelli feedback alone could enable flights in proximity to the ground indefinitely. In practice, this would be limited by battery life and the presence of obstacles.

Figure 8.

Altitude control and upright stability using only onboard feedback. The ocelli provided feedback to maintain upright while the aerodynamic ‘ground effect’ maintained altitude. During the first 0.3 s of flight during take-off, the lift was slightly larger, after which it was reduced by a small amount. Altitude was maintained by the effect that lift slightly increases as the vehicle comes in proximity to the ground. (Online version in colour.)

6. Discussion

In this work, we showed how an ocelli-inspired vision sensor can be used to stabilize the upright orientation of a fly-sized flying robot. Although a number of sensors have been proposed that could, in principle, be integrated into such a vehicle [44,53,54], few have been operated on a fly-sized robot [47], and none have previously performed in free-flight. Hence, these results constitute the first demonstration of onboard sensors in free-flight at this scale. We remark that even though the vehicle uses a wire tether to provide control commands and returns feedback, it does not augment stability (see §2.2). Power autonomy is explored in separate, but parallel efforts [59,60]. Our key contribution is a demonstration that angular velocity feedback from the ocelli sensor can stabilize the pendulum-like dynamics of our vehicle. By responding only to the angular velocity of the light source rather than its position, the ocelli-output feedback controller proposed here can stabilize the vehicle even if the light source is not directly overhead.

The operating range for the light source of ±60° from vertical could be extended by choosing sensors that integrate luminance over a larger field of view and orienting them closer to the horizon, or by ‘triangulating’ the direction d of the light source. The analysis given by equations (4.1)–(4.5) shows how the number of sensors could be reduced to three to reduce weight, thereby mimicking the number found on insects. The computational requirements for estimating angular velocity on our robot were minimal and would represent a negligible component of a flight controller. Processing required approximately 1 million floating-point operations per second at 10 KHz for filtering and optional polynomial evaluation, but this was over-engineered to ensure fidelity. Simulations indicate that these requirements could be dramatically reduced without sacrificing performance by reducing filter order and update rate.

It may be possible for the ocelli to estimate and control motion around the body z-axis as well. One possible approach is to use approaches inspired by optic flow such as autocorrelation [61], which requires a nonlinearity [62], in contrast to present analysis which is linear in light input. Ambulatory flies use their ocelli to orient towards a light source [63], suggesting they may incorporate such a control strategy. In a higher-level flight scenario in which other states such as lateral position [33] are controlled, the high bandwidth and few inputs of the ocelli are well suited to a fast inner loop to stabilize attitude. For example, in [33], motion capture provides an estimate of ω used for stability; this could be replaced by the estimate derived from the ocelli sensor.

The results presented here also support a hypothesis for how the ocelli could function to stabilize many flying insects. Fluid mechanics models based on quasi-steady approximations have suggested that many flying insects that hover, including moths, locusts, beetles, flies and honeybees, also have an instability similar to our vehicle [28–31,64,65]. Like our vehicle, the basic structure of these animals consists of a pair of wings (or two pairs that are joined) and a body that hangs below them [51]. In the case of the fruit fly Drosophila melanogaster, it has been suggested that they could achieve stability using feedback from the halteres [30,31], which measure angular velocity [41]. This has extended to pitch stability in other species in [32]. In the aforementioned studies [28–31,64,65], however, conclusions are based on simulations that approximate the unsteady flow phenomenon generated by flapping-wing flight with a ‘quasi-steady’ model [66]. This could neglect important effects. It is therefore desirable to perform a more detailed characterization under the conditions of coupled translational and rotational motion that constitute this instability. To replicate this motion in a dynamically scaled fluid mechanics model would require a six degrees-of-freedom articulated device under a ‘captive trajectory’ to simulate its inertia in a large tow tank [57]. This has not yet been achieved. Using a computer instead to perform computational fluid dynamics simulations would require a large fluid volume that could require a prohibitive amount of computation power [67–69]. Another approach is to carefully manipulate sensory feedback available to alert, behaving animals [70,71]. However, precisely controlling sensory input during flight is difficult because of the small size of the animal. In addition, multiple modes of sensory feedback are typically required to stay aloft [3], making it difficult to isolate the effects of a single sensory organ. Hence, our robotic fly constitutes a new method to probe insect fluid mechanics that avoids these difficulties.

Our results, particularly those in figure 5, support the view proposed in [30–32] and others that angular velocity feedback or ‘rate damping’ is sufficient for upright stability in Drosophila and similarly shaped species. While the robotic fly and these vehicles are not identical, the approximations made in the model given in §3 apply equally to both the robot and these animals. One difference, however, is that in these animals the body hangs diagonally [32,51] rather vertically as in our vehicle. In both cases, however, the CM is centred below the wings. For these animals, the inertia matrix has off-diagonal xz terms that couple torques in the x-direction to motions about the z-axis and vice versa, but it can be shown that this does not alter the stability of the system. To provide a particular example, we consider the honeybee. In table 2, we give the equivalent parameters for the honeybee Apis mellifera, which has similar size and weight to our vehicle. These were derived from measurements and calculations in [32,51]. Drag on the wings is higher probably because of the higher typical flapping frequency of bees (197 Hz) compared with our vehicle (120 Hz), but the moment arm r_w is much shorter. In the case of the honeybee, the model suggests that the minimum necessary damping gain to achieve upright stability is similar to that of our robotic fly (table 2).

Table 2.

Estimated parameters for the honeybee Apis mellifera.

quantity	symbol	quantity	units
mass	m	102 × 10−6	kg
moment of inertia (y-axis)	J2	2.2 × 10−9	kg m2
wing drag factor	bw	4.4 × 10−4	Ns m−1
distance, CM to wings	rw	3.3 × 10−3	m
minimum gain (equation (3.9))		7.8 × 10−8	N ms

In insects, studies have suggested that the ocelli mediate a fast, derivative-like reflex that acts in corrective flight manoeuvres. Neural recordings of the large L-neurons in the locust show a phasic, rate-dependent response [36], and in flies the ocelli mediate a head rotation reflex that turns it towards sudden changes in light direction [37]. This reaction happens with significantly less time delay than the compound eyes [52]. Hence, the direction, speed of response and rate dependence of the ocelli reflex are consistent with a rate-damping flight stabilizer. Studies have shown that honeybees can nevertheless fly with their ocelli occluded by opaque paint, but flight precision and direction is significantly disrupted [72]. Recent results in the blowfly Calliphora indicate that the eyes and ocelli act in concert, with the ocelli providing a faster but more crude estimation of rotation, whereas the eyes respond with a more precise estimate at a slightly later time [39]. Hence, while, theoretically, the ocelli can provide the necessary damping for these animals to stay upright, in practice, the reflex may be mediated by a superposition of feedback from both the ocelli and the compound eyes. It is not currently known how the honeybee, which does not have the gyroscopic halteres that flies use to estimate their angular velocity, stays upright [20]. Our results are therefore consistent with a view that they could stay upright using an angular velocity reflex mediated in part by feedback from the ocelli that is functionally equivalent to the controller described here.

Funding statement

This work was partially supported by the National Science Foundation (award nos. CCF-0926148 and CMMI-0746638) and the Wyss Institute for Biologically Inspired Engineering.