Dynamic Facial Prosthetics for Sufferers of Facial Paralysis

Abstract

Background

This paper discusses the various methods and the materials for the fabrication of active artificial facial muscles. The primary use for these will be the reanimation of paralysed or atrophied muscles in sufferers of non-recoverable unilateral facial paralysis.

Method

The prosthetic solution described in this paper is based on sensing muscle motion of the contralateral healthy muscles and replicating that motion across a patient's paralysed side of the face, via solid state and thin film actuators. The development of this facial prosthetic device focused on recreating a varying intensity smile, with emphasis on timing, displacement and the appearance of the wrinkles and folds that commonly appear around the nose and eyes during the expression.

An animatronic face was constructed with actuations being made to a silicone representation musculature, using multiple shape-memory alloy cascades. Alongside the artificial muscle physical prototype, a facial expression recognition software system was constructed. This forms the basis of an automated calibration and reconfiguration system for the artificial muscles following implantation, so as to suit the implantee's unique physiognomy.

Results

An animatronic model face with silicone musculature was designed and built to evaluate the performance of Shape Memory Alloy artificial muscles, their power control circuitry and software control systems. A dual facial motion sensing system was designed to allow real time control over model – a piezoresistive flex sensor to measure physical motion, and a computer vision system to evaluate real to artificial muscle performance.

Analysis of various facial expressions in real subjects was made, which give useful data upon which to base the systems parameter limits.

Conclusion

The system performed well, and the various strengths and shortcomings of the materials and methods are reviewed and considered for the next research phase, when new polymer based artificial muscles are constructed and evaluated.

Background

Breedon and Vloeberghs investigated the use of Shape Memory Alloys (SMAs) as dynamic facial prosthetics for sufferers of facial paralysis.¹

Facial paralysis without chance of spontaneous recovery can occur in a patient following a stroke, tumour or trauma from accident or various congenital issues such as Moebius syndrome and muscular dystrophy. A common symptom is one half of the face is left with little or no movement. The underlying problems tend to stem from damage or maldevelopment of the facial (VII cranial) nerve, though on occasions such as damage from accident, the issue can also stem from the non-function of the muscles themselves. Beyond aesthetical issues, problems associated with facial paralysis can be split into two basic groups: physical and communicative.

Physical symptoms can include drooping of the corner of the mouth, which leaves the sufferer unable to prevent drooling. The inability to blink can mean the cornea can become dry, potentially leading to blindness if left untreated. These problems can be solved using passive surgical methods, such as pulling the corner of the mouth up to a neutral level using a gold wire, or weighting the eyelid down with a small gold weight inserted beneath the skin. While these passive solutions lead to an improvement, it is arguable that a more active solution would be preferable. An extreme active solution involves the transplantation of muscle tissue from the groin (gracilis muscle) into the cheek, though this is limited by implant extrusion, donor site morbidity, and surgical candidacy.²

Facial nerve grafting, cross-face jump grafts (splitting the signal of the functioning facial nerve to both sides of the face), or anastomosis i.e. reconnecting a previously branched nerve with other cranial nerves, for example the hypoglossal-facial nerve have been evaluated. The latter affords the patient an ability to move the face voluntarily by tongue thrusting.³ These have varying levels of success, depending on candidate's age (preferably young), level of atrophy of the paralysed muscle (more than two years will make it unlikely to function again) and also on what nerves are still functional – this is particularly an issue following stroke induced paralysis.³

While the above methods lead to verifiable results, it is the author's belief that the implantation of a fully encapsulated, electrically stimulated artificial muscle would lead to a more versatile solution. These include a wider surgical candidacy range, shorter rehabilitation time and greater control over the numerous subtle facial expressions.

Communication is an area that is hugely affected by facial paralysis. Nonverbal communication via facial expression plays a vital role in conversation. Emphasis for punctuation and questions are placed via movement of the ‘medial’ and ‘lateral frontalis’ (forhead) or ‘corrugator’ (eyebrow) muscles. Whilst listening to someone else speak, movement of the ‘zygomatic major’ (cheek) or contraction of the ‘orbicularis oris’ (eye) can imply understanding and attention. Perhaps the most universal expression or ‘emblem’ is the smile, which can infer numerous emotions such as joy, embarrassment, uncertainty, sympathy, contempt or compliance.⁴ As Ekman points out there can be eighteen classifications of smile, but only one accompanies spontaneous positive emotions.⁵ With this in mind, the smile has been set as the major focus of the research thus far, with emphasis on capture, understanding and recreation of the nuances in varying types.

The areas of the face that move synchronously during an emotive expression have required an in depth investigation. Of particular interest and one of the main sources is the "Facial Action Coding System" (FACS).⁶ This study takes the muscle groups and classifies them according to their timing and natural tendency to act together during a particular facial expression.

The various movements are divided in "Action Units" or AUs, of which there are forty six specificity related to underlying facial muscle contraction and their effects on the skin and subcutaneous fascia.

The following action units have been primary in relation to creating a realistic smile (see Figure 1):

Figure 1. Diagram of Action Units attributed to the creation of an involuntary "Duchenne Smile", alongside some of the more common furrows that appear.

• AU6 - Cheek Raiser – Lower orbicularis oculi contracts

• AU12 - Lip Corner Puller – Zygomaticus Major contracts

• AU25 - Lips Part - Relaxation of orbicularis oris or depressor labii inferioris

While FACS concentrates on the activity of the underlying facial muscles from a perspective of surface feature displacement, anatomical studies of the maxillofacial region are also taken into account. Alongside published data on facial movements, specific data was gathered on displacement, timing and the force required to move various regions of the face.

Method

Before the artificial muscles could be constructed, an understanding of the motion of lips - both duration and displacement of a smile, along with the force required for moving lip corners had to be garnered. Data was collected from a number of subjects performing smiles of differing intensities by analysis of video recording. This was done to establish the movements and displacements of smiles in faces of various age, shape, nationality and sex. The subject's expressions were recorded on video, then each frame normalised for uniformity. This gave a starting point to measure the timings and other motion parameters required to activate artificial muscles. A study of the forces required was then made by videoing the displacement of the lips and cheeks using a high accuracy spring balance attached to the lip corners. The specific displacement was calculated by overlaying a grid on the individual video frame when the balance showed specific applied forces.

A device was constructed using a cascade of SMA wires attached to a silicone musculature in order to recreate some of the general actuations associated with smiling. The rationale was to create and test algorithms which infer facial expressions, and recreate them in the animatronic head via a minimum number of sensors. The SMA cascade was placed approximately beneath the temple area of the model head, and the silicone muscle representations were attached to the cascade using monofilament thread. While not wholly realistic in terms of where such a device would be placed if it were to be implanted in a real patient, it allowed testing and evaluation of indirect actuation on the lip corners and lower lip.

An actuation was performed to the lower eyelid area when a smile intensity greater than 50% was requested. This arbitrary percentage can be considered the point at which lower Orbicularis Oris (AU6) movement is seen to occur in a spontaneous smile.

Artificial muscle movement was realised via either one of two different control methods – first a piezoresistive flex sensor was used to control the displacement of the muscles in proportion to the amount of bend it was subjected to. Implantation of flex sensors is one of the possible methods that could be used to sense the healthy muscles. This is expanded on further in the discussion section.

Secondly a computer vision (CV) facial expression recognition system was written. The rationale for building a CV system is one of configuration and calibration of the artificial muscles. Similar to a biological muscle following implantation, an artificial muscle would require some form of physiotherapy in order to actuate correctly according to the individual's physiognomy. It was decided to write computer vision algorithms with integrated machine learning to perform this automated therapy. The symmetry of real to artificial muscle actuations were assessed, and the artificial muscle parameters adjusted accordingly as asymmetries are detected. As a consequence, a correlation of sensor readings to motion is built up, which refines and customises the device to the individual wearer.

A program and interface was written to utilise facial recognition and feature tracking primitives from the Intel OpenCV⁷ and Machine Perception Toolbox (MPT)⁸ opensource CV libraries. The program could recognise when a face was in front of the camera, detect the position of the eyes and mouth, and calculate the displacement of the lips as they moved. This was then translated into a measure of how much the subject was smiling. This arbitrary smile intensity could refine itself to an individual user as they continually used the system. The detection algorithms made use of a combination of Canny Edge Detection (Edge Orientation Histogram) and Box Filters (Viola-Jones)⁹ to calculate a position and shape of the lips throughout the smile. Adaptive boosting (Gentleboost implementation) was applied to classify and progressively add new components to a frontal face Haar Cascade¹⁰, which allowed the program to refine itself towards a specific face or subject.

Power control circuitry was designed and built which allowed a microcontroller to strain individual SMA muscle strands, via controlled pulse width modulation (PWM). The advantage of using PWM was one of finer tuned position control, and an ability to hold an expression in one place for an indefinite period.

Results

The video analysis of smiling was carried out only on smiles that were deemed to be spontaneous. This was because there is a reported non-linear relationship between duration and amplitude in the onset phase of posed smiles.¹¹ The sequenced still shots were normalised (rotated and moved) to line up the inner eyes as best as possible, as an out of plane movement – predominantly a twist of head to left and pitch upward and backward was found to occur frequently when spontaneous smiles were involved. This was particularly the case when the subject got close to breaking into laughter. This reduced the dataset significantly to eight accurate sequences. Of the sequences that remained, the right and left lip corner were represented as a mean displacement of the corner points. The average duration and displacement of the recordings are detailed in Table I

Table I: Measured values of spontaneous smiles.

	Mean	SD
Duration (secs)	1.3	0.9
Displacement (mm)	0.707	0.65

Having broken the video sequences into 66ms frames which could be analysed in step form, a 10mm grid was overlaid. Guide dots were drawn on each face before filming in order to calibrate the grid with the face being analysed. This also helped to keep track of individual points as they moved throughout the duration of expression.

It was discovered that head movement was a limiting and confounding factor when collecting large volumes of data and that it caused a large standard deviation. Rather than use the measured data it was decided to use data from the standardised expression video databases - the Cohn-Kanade DFAT,¹² detailed in Table II.

Table II: Cohn-Kanade values of spontaneous smiles.⁸.

	Mean	SD
Duration (secs)	0.52	0.32
Amplitude (change in radius) (mm)	0.05	0.07
Ratio of duration to amplitude	17.96	13.49

As well as manually tracking, some automated supervised corner tracking systems were also used. (Voodoo Camera Tracker¹³ and EyesWeb¹⁴). A feature of both programs was an ability to export CSV files containing all the motion tracking information. While this was primarily designed for mapping video sequences onto 3D meshes in animation programs such as 3DS Max or Bryce3D, though in this case it was imported into a MySQL database for analysis. The main problem found with this was the sheer number of tracking points (see Figure 2 for example). Efforts had been made to introduce specific "points of interest" (PoI) on the face with a coloured marker before filming, but even still the tracker program chose a large number of corner PoI in the eyes and hair. Before running the program, these features were removed from the video sequence. As can be seen in Figure 2, the automated tracking created an interesting overview of the timing of non-lip corner areas and their movement throughout the smiling sequence, but stricter supervision of tracking points must be implemented before implementable data can be gathered from this method. One point to note is the slight asymmetries of movement on either side of the face. This tended to vary person to person, and expression to expression.

Figure 2. Normalised frames taken from smiling sequence. Green tracking signifies lack of movement from one frame to the next, whereas red signifies movement. Each frame represents 66ms (15FPS).

The model silicone musculature (Figure 3) that was built was laser etched until it had an elastic resistance approximating the measured values of real facial muscles (approximately 1N of force was required to displace the "lip corners" by 0.8mm. 0.6N was required to move the lower eyelids by 0.4mm). A caveat for this collected data was it source was that of healthy muscle tissue, rather than partially (or fully) atrophied muscle. Nonetheless, it has given a base level from which to work.

Figure 3. Animatronic model with silicone musculature.

To achieve a movement in the SMA artificial muscle from neutral to apex in less than one second, the required electrical power was 12W (12V at 1.0A) for lips and 2.1W (6V at 0.35A) for eyes. As this movement was caused by heating rather than potential difference there was no way to affect the discharge or offset phase speed. This was found to be slow: over a 1.5 seconds minimum, depending on how long the expression had been held. Due to variations from hysteresis and the non-linearity in SMA, figures for the timing of actuations are not suitable to be tabled.

Discussion

The facial and smile recognition by the computer vision system gave quite satisfactory results and the artificial muscles responded to commands by the software and caused the animatronic head (Fig 3) to smile in response to testers and visitors smiles over the course of a week's testing. The system occasionally had trouble calculating smiles for those with darker skin, or when facial hair closely matched to colour of the subject's skin. In some cases a smile had to be exaggerated in order to register. When a person was new to the system, registration would be jumpy until the cascade began to recognise or ‘learn’ their features. This aspect is not considered negative, as the system must train itself to its subject, and will refine itself with continual use.

While SMAs show a lot of promise regarding their actuation strain and force, the investigation showed that they have relatively high power consumption, relatively slow response time, hysteresis, and are susceptible to changes in ambient temperature. This suggested that much further work was needed to find and explore other suitable candidates for use within an in vivo device.

The investigation and testing of materials most suitable for the creation of the next iteration device should focus on prerequisites such as low-power consumption, biocompatibility, affordability and controllability. Polymers have been developed that respond to numerous varying stimuli with a change that can be either temporary or permanent. The stimulation sources include heat, light, chemical (pH), pressure, magnetic and electric field.¹⁵ Many of these sources are unfeasible for in-vivo stimulation, so only those materials which are electrically or heat activated have considered and/or fabricated.

Electro active polymers (EAPs), consisting of various classes of conjugated and ionic polymers, along with dielectric elastomers are a relatively new class of material. They are only recently at a stage of being well modelled and documented in terms of their mechanical efficiency, physical constraints, stability and best fabrication practice. EAPs fall into two broad categories – ‘dry’ or electronic and ‘wet‘ or ionic.

Within the dry category there are a number of sub classes such as piezoelectric, electrostrictive and dielectric elastomer. These polymers change shape or dimension due to the migration of electrons across an applied electric field.¹⁶ The term ‘dry’ refers to the fact that the material can actuate in air, or without the presence of an electrolyte. They tend to exhibit large strain and actuation force, and can hold their position without back relaxation under an applied DC field, but generally require very high voltages to activate. This is not necessarily a problem as their current consumption is in the micro-amp region, so can be easily isolated if encapsulated in medical grade silicone. The advantage of dielectric electro-active polymers (DEAP) is that they can exhibit performance equal to or exceeding natural muscle in all important metrics – stress, strain, work density etc. This comparison is shown in Table III ^17,18 No other solid state actuator can achieve statistics to match natural muscle in this way.

Table III: Comparison of Muscle to Dielectric ElectroActive Polymers^17,18.

	Natural Muscle	DEAP
Maximum Strain (%)	> 40	120-380
Max Exerting Pressure (MPa)	0.35	3.2 - 7.7
Work Density (KJ / m3)	1037	960 - 1100
Min speed of full contraction	< 1 sec	< 100 msecs

Some efforts have been made to test DEAP within the face, particularly Tollefson and Senders proof of concept artificial "blinking" muscle.¹⁹ This involved electrically isolating EPAM™ (Artificial Muscle Inc.), via encapsulation, then connecting the resultant muscle to a pair of inert expanded Polytetraflouroetheylene (ePTFE) ‘slings’ – one above and one below the eye. Contraction of the EPAM caused the eyelid to close. While this experiment was conducted on cadavers, SRI, the parent company of Artificial Muscle Inc. claim they have been testing bio-compatiblility of EPAM in live gerbils. [currently unpublished]

The other interesting option comes from the ionic class of EAPs. These are divided into three main classes: Ionic Polymer-Metal Composites (IPMC), Conjugated Polymers (CP), and Ionic Gels. These actuators change their shape due to the movement of diffused ions which are encapsulated in the membrane or gel. All three are soft and flexible, biocompatible and have low power consumption (normally less than 1W). There are numerous ways to fabricate each type - with varying success. The ionics do suffer some drawbacks and face challenges if they can be considered viable – often they produce quite a low blocking force, can be difficult to maintain their static position, or are chemically sensitive - thus require sealing against contamination of the ionic content. This encapsulation can result in reduced performance efficiency. IPMCs are expensive to produce, requiring a relatively large quantity of a noble metal such as gold or platinum for production. Conductive Polymers still perform best when immersed electrolytes such as tetrabuthylammonium perchlorate (TBAP) which even at the low concentration of 0.04M, is non-biocompatible.

Currently DEAP and IPMC are being further researched by the authors. Currently IPMCs (see Table IV) are undergoing testing, both in terms of fabrication improvement and integration. DEAP is at the beginning of fabrication research. As can be seen from Figure 4, the forces exerted by the current IPMC samples are extremely low, and tip displacement reduces significantly when cycled in dry air. The exerted forces can be improved significantly by using a thicker actuator, but this comes at a significant reduction of reaction speed. (Increasing to seconds for a full cycle)

Table IV: Comparison of tested IPMC samples.

	Surface Plating	Width (mm)	Length (mm)	Thickness (mm)	Dry Weight (g)
1	Palladium	9.1	33.2	0.11	0.0683
2	Silver	5.5	32.6	0.19	0.0815
3	Silver	5.7	24.2	0.18	0.0614
4	Palladium	5.5	23.6	0.19	0.0595
5	Gold	7.0	41.7	0.20	0.0735
6	Palladium	5.5	25.7	0.20	0.0635

Figure 4. Measured blocking forces of IPMC samples varying in size and surface plating, subjected to 3V at 0.5Hz.

While there are many positive aspects to dry and wet EAPs, both technologies have issues that first must be overcome. DEAPs with their extreme voltage requirements (albeit whilst drawing virtually no current) are unconfirmed in terms of biocompatibility. Any device designed will require years of clinical trials to be proven safe. Even though Artificial Muscle Inc (AMI) have been investigating how EPAM reacts in live animals, they have released no findings after two years. It can only be speculative as to what this might mean.

On the other hand, EAPs have a huge number of advantages over all the other actuator technologies. They are relatively cheap and the various procedures required to fabricate are moderately simple. The reaction time is extremely fast – in the kHz range of frequency response. A blocking and actuation force that equals or exceeds those of natural muscles, and is easily shaped into whatever form is required, alongside an excellent level of compliance. In some respects, control of the ‘muscle’ is somewhat counter intuitive, as it requires power to maintain the neutral relaxed expression. Contraction of the muscle would involve reduction of potential across its electrodes. The actual layout and shape of EAP muscle is currently being experimentally refined. As the most common base polymers come commercially in tape form (a silicone based tape by NuSil and an acrylic ‘VHB’ tape by 3M) the actuator must be designed in layered two dimensional planes. This is contrary to something like an IPMC bending actuator with a standard linear form, shown in Figure 5.

Figure 5. IPMC cantilevered strips in relaxed (a) and charged (b) arrangement. Render of linear chain actuator shown (c).

As a contrast to DEAP, IPMCs are almost inverse in terms of their strengths and shortcomings. Foremost in their favour is verified bio-compatibility, but also the ability to be liquid cast into any shape, thickness and configuration required. On the other hand, to be realistic in terms of facial muscles, their blocking force to speed ratio must be addressed.

The technology for the sensing movement of the contralateral healthy muscles is in a mature state of development. It is a necessity to be as non-invasive as possible while gathering the maximum information on the timing and distance required. The previous paper on this topic [1] suggested electromyography (EMG) as the chosen method of sensing, though it is equally permissible to envisage implanted flex sensors as a viable solution. Through testing it seems most likely that a combination of both will give the best results: EMG is required to confirm that a deliberate muscle movement is taking place, then the flex sensor would be used to measure the magnitude and precise area of actuation.

The flex sensing element could be piezoresistive conductive polymers such as a bi-layer composite, consisting of a compliant polymer films coupled with crystals of molecular conductors. These consist of ion-radical salts (IRSs), typically based on tetrathiafulvalene (TTF) derivatives "such as bis(ethylenedithio) tetrathiafulvalene (BEDT-TTF), which exhibits dramatic changes in conducting properties under isostatic pressure or uniaxial strain".²⁰ These bio-compatible all-organic sensing devices have been prototyped by Laukhina et al into a device for monitoring a person's breathing rhythm. In the same paper they discuss the possibility of using these sensors for detection of tissue movements. They exhibit high sensitivity (sensing changes in pressure down to single mbar range) while exhibiting a linear change in resistance. Alternatively piezoelectric thin films such as polyvinylidene fluoride (PVDF) may be used. PVDF is a mature and well documented technology, which is easily integrated and commercially available

As the system is being designed to be entirely encapsulated within silicone pouches, biocompatibility should not be a problem. Nonetheless some of the most promising materials being proposed for use within the pouches are currently uncertified by medical regulators. Potentially this may result in a delay of years before the system can go into commercial production. Every effort is being made to keep the construction materials cheap enough for this device to remain affordable for the estimated 250,000 per year sufferers worldwide of permanent facial paralysis.³

Conclusion

Investigation of a wide range of materials and methods, along with a surface level investigation of facial anatomy and facial expressions has shown that an active prosthetic solution for facial paralysis requires more than a set of simple actuation devices. The speed, timing and power consumption are just a few of the many considerations. It is clear from facial anatomy and expression research that an actuator with multiple degrees of freedom, with independently addressable ‘actuation zones’ is required to recreate even the most basic proto-expression such as a smile. The maximum size/area of the artificial muscle is severely limited when correlated to the amount of strain that is required, especially in relation to the actuation stress that must be exerted. Stacking of thin film actuators should help to overcome this limitation.

The study of the Facial Action Coding System has helped demonstrate how many variables could potentially be involved if the muscle is to recreate multiple expressions. With this in mind, further development shall still concentrate on the creation of a limited number of convincing movements such as smile and squint, rather than aim for a universal expression set. As the basic movements are convincingly recreated then further expressions shall be added to the muscles ‘repertoire’.

The materials to be used are still debatable – every actuator type discussed has its own strengths and weaknesses. Much emphasis has been placed on IPMCs within the research so far, as it seems they are the most likely to provide a stable long term solution. That stated, their relatively embryonic stage of development in terms of published fabrication methods and control mean they may not provide a mature enough technology to proceed directly with in the next stage of prototype. Their speed to force ratio must be addressed. Achieving a reduction of production cost must also be addressed. A combination of IPMC and Conductive Polymer actuators such as the sulphonated polystyrene (sPS) may achieve good results and provide a pathway towards addressing these concerns. In the mean-time, it seems most viable to concentrate on dielectric EAPs as a next prototype stage. Even if it subsequently transpires that their high voltage requirement creates bio-compatibility issues, they are currently the only compliant polymer actuator that matches almost all the performance metrics of natural muscle while retaining a low power consumption and price. Their robustness and longevity is perhaps the only major issue that must be addressed.

In terms of sensing, both methods discussed (flex and EMG) are viable, and are in a position to only require integration rather than amelioration. Further development may find that a combination of the two discussed yield the most reliable results.

The first physical prototype which was developed and discussed in this paper has helped to confirm or disprove a number of assumptions that were taken early in the investigation. The use of non-compliant cascade actuators meant indirect actuation of the muscle was necessary. This led to difficulty in rigging the muscles, and an inability to push as well as pull. It is felt that every subsequent iteration should focus on an in-situ design which will morph in shape to create actuation rather than indirect pulling on the nolonger functional muscle, in an effort to create expression on the face.

The use of shape memory alloys has confirmed two of their biggest downsides – namely power consumption and brittleness over time. Their 10W (average) power consumption would not be sustainable from an implanted battery such as that of a pacemaker. Constant indirect resonant induction charging from an external power-pack would be an option, but it is the authors belief that the more efficient actuators with movement caused by either ionic transfer or Maxwell stresses would be a much better option.

The ideal situation for an eventual design would be to have all power and control signalling to the device inductively telemetered through the skin, preventing the need to embed digital signal processors or power sources. This is eminently possible with current commercially available technologies.