A third eye for the surgeon

Neurologists and engineers have developed a new camera system for surgeons, which looks where the eyes look. This innovative device uses voluntary and reflexive eye movements, which are registered in 3‐D by video‐oculography and then computed online, as signals to drive the camera servo motors in three planes: yaw, pitch, and roll (fig 1). Its primary objective is to allow a freely mobile user to aim the optical axis of a head mounted camera system at the target(s) at which he/she is voluntarily looking in the visual field, while the ocular reflexes stabilise any image shaking by naturally counter rolling the “gaze in space” of the camera during head and visual scene movements and during locomotion. The biological system of the human ocular motor control incorporates these functions, but so far, no camera system has combined free user mobility with image stabilisation and unrestricted exploration of the visual surround.

Figure 1 The user is wearing a head mounted device equipped with video eye trackers. They consist of two laterally attached infrared video sensors and two infrared mirrors, one in front of each eye, which reflect the images of the eye to the sensors. The image information is transmitted to a video‐oculography computer, which uses image processing algorithms to determine the three dimensional eye positions. The eye position is transformed into a signal for servo motors that control the optical axes of a rotatable head fixed camera system. Because the infrared mirrors of the eye movement detectors are transparent to visible light, the user's field of view is not restricted. Thus, the axis of the “third technical eye” is always aligned to the axes of the user's eyes. Consent was obtained for publication of this figure.

As biological and technical systems are governed by closely related physical laws, their sensorimotor control mechanisms are similar. The biological principles that have evolved over millions of years continue to serve as inspiration for new technical constructions. Here the human ocular motor system is exemplary. Visual exploration is made possible by the use of voluntary saccades when the eyes are quickly moved from one visual target to another, and voluntary smooth pursuit, when the eyes follow a moving target. The vestibulo‐ocular reflex relies on inner ear sensors that signal both linear accelerations and angular head velocities to the brain. This velocity information is transformed mathematically by integration into a positional signal, which is then inverted and delivered to the eye muscles.¹,² During head movements, the biological reflex moves the eyes in their orbits in the opposite direction to the head motion, thus ensuring a stable projection of the visual scene onto the retina. A similar mechanism becomes active when large field visual stimuli are moved in front of an observer (optokinetic reflex).

This complex ocular motor system also controls the orientation of the optical axis of the novel head mounted camera. To prevent perception of apparent motion of the visual scene (oscillopsia) during rapid eye and camera movements, an artificial saccadic suppression mechanism can be incorporated, which is triggered by saccade onset. This artificial motion suppression can be achieved by repeating (“freezing”) the last frame acquired before saccade onset for the duration of the camera saccade. Depending on the saccade amplitude, the image has to be frozen for about 100 ms.

Thus, the surgeon using this camera can move his/her head and eyes during image acquisition without having to fear image shaking artefacts, while at the same time continuously document an operation. In addition, the vergence angle of the eyes delivers valuable information for a possible autofocus functionality, as the vergence angle directly depends on the distance from the observed object.

There are numerous commercially available head referenced scene cameras combined with an eye tracker that highlights the visual target in the scene image. However, all of these systems possess a head fixed camera that does not move with the eyes. Their field of view is restricted by the optics, and therefore the visual target is not always in the range of the scene camera. In contrast, the novel system described here covers the whole explorable field of view and therefore the camera documents every eye movement of the surgeon, for example, when he/she is looking down. The image centre captured by the scene camera coincides with the surgeon's foveal fixation, assuming there is accurate calibration of eye tracker and scene camera. The current system has an accuracy of 0.45 degrees, which is well within the 2 degrees of the foveal area.

Several medical and non‐medical applications are conceivable for the camera during free locomotion:

• analysis of the ocular exploratory behaviour of freely mobile test persons or patients, for example, to assess the development of gaze control from childhood to adulthood or abnormal visual exploration in psychiatric or neurological disorders,

• animal films that correlate behaviour with visual control,

• sports newscasts showing the viewpoint of the sportsman in action,

• analysis of stimuli that control gaze for advertisement purposes,

• night vision tasks in which the user enjoys mobility while preserving his field of view and continuing watching, for example, a thermal picture projected onto the half transparent infrared mirror.

Acknowledgements

The research project in which the camera was developed was supported by a grant of the Bavarian Research Foundation and was evaluated by an international group of experts, neurophysiologists, and engineers. We are grateful to J Benson for critically reading the manuscript.