Presenting a model for dynamic facial expression changes in detecting drivers’ drowsiness

Abstract

Drowsiness while driving is a major cause of accidents. A driver fatigue detection system that is designed to sound an alarm, when appropriate, can prevent many accidents that sometime leads to the loss of life and property. In this paper, we classify drowsiness detection sensors and their strong and weak points. A compound model is proposed that uses image processing techniques to study the dynamic changes of the face to recognize drowsiness during driving.

One of the problems of modern society is road safety and security, and sleepy drivers are a major cause of accidents on roads around the world (1). Fatigue and the monotony of driving are two important factors that can lead to a decrease in a driver’s alertness level (2). In a report released by the National Highway Traffic Safety Administration (NHTSA) in 1994, Knipling and Wang determined that, from 1989 through 1993, an average of 100,000 crashes occurred annually due to drivers’ fatigue and drowsiness. In these crashes, 1,550 people were killed and 40,000 others had various injuries (3). From an economic perspective, America’s sleepy drivers cost the economy about $12.4 billion annually (4, 5). In Britain, driver fatigue is one of the main contributors to 20% of the accidents that occur (6). In addition, driver fatigue is the main contributor in 18% of fatal accidents in Australia (7). Research that was conducted in Japan (8, 9) and France (10) indicated that drowsiness is the main cause of fatalities in vehicles. There are no exact studies or statistics that address this issue in other countries (11). Generally, driver fatigue is a major cause of 25% of the traffic accidents that occur, and approximately 60% of traffic accidents result in death or serious injuries (12). According to a report produced by the World Health Organization (WHO) in 2011, Iran has more traffic accidents than any other country, and Iran ranks fifth in the number of deaths caused by traffic accidents (13). According to a report released by WHO, the number of people killed in traffic accidents in Iran is about 44 per 100 thousand people (13). According to a UNICEF report, the accident rate in Iran is 20 times more than the global average, and it is the major cause of the death of children (14). According to the report, a person is killed every 19 minutes on the roads in Iran, and a person is paralyzed for life every two minutes by the serious injuries inflicted by traffic accidents. In addition to the death and human suffering, traffic accidents in Iran cause $6 billion in damages to the economy annually, which is more than 5% of Gross National Product (GDP). It should be pointed out that about 90% of the traffic accidents in Iran occur among low-income or middle-income people. According to figures released by Iran’s Office of Traffic Safety and Road Maintenance and Transportation, traffic accidents in the country killed 20,068 people in 2011. This number was comprised of 5888 deaths in urban settings, 12,232 deaths in suburban settings, 1803 deaths in rural settings, and 145 deaths in unknown settings (15). In addition to those who were killed, 297,257 people were injured, 216,207 of whom were males and 81,050 were females (15). Sleep disorders can lead to fatal traffic accidents in two ways, i.e., 1) the driver goes to sleep, making an accident virtually certain and 2) the driver is drowsy, is not alert, and cannot deal with traffic issues in a timely manner (13).

In general, methods for detecting drowsiness are classified in one of two ways, i.e., the supervisory-based method and the vehicle movement-based method. In the supervisory method, the main means of detecting drowsiness are sensors and cameras that record the driver’s physical symptoms. Then, the signals from the sensors and cameras are sent to a computer that processes them and identifies the driver’s level of alertness. It should be noted that the methods associated with the supervisory approach are more developed and have more diversity than the second category because the mathematical and engineering analyses are comparatively easy. However, drivers can tire of the nuisance of measuring devices, and the cost associated with the cameras and other equipment makes them prohibitively expensive for general use. The driver’s drowsiness can be categorized by three different types of sensors, i.e., a physiologic sensor, a performance sensor, or a vehicle response sensor. Among these three methods, the best method is the one that is based on human physiological consequences. It is possible to measure changes in various physiological signals, such as EEG, ECG, and EOG. Other vehicle-response sensors, driver-performance sensors, and their advantages are stated in Table 1.

Table 1.

Common methods of evaluating drowsiness and their advantages and disadvantages

Method name	Advantages	Disadvantages
Based on physiological measures (EEG)	By using brain waves, drowsiness can be efficiently and accurately detected.	It is not realistic, because to get these signs, electrodes must be attached to the body, which is unpleasant or annoying to drivers.
Based on vehicle measures	Lane tracking, vehicle steering wheel changes, the number of lane crossings, and the distance from the front vehicle can be used in detecting.	Having restrictions against some changes, including vehicle type, driver experience, road topology, road quality, and ambient light; in addition, the processing of these methods requires considerable time to analyze the drivers’ behaviors that cause them to be unaware of micro-sleep.
Based on behavioral measures (image processing)	In drowsiness, sensible changes can be seen in appearance and face of people, and the most important changes are in the eyes, head, mouth, and sitting posture. By taking a picture of the driver and using image processing techniques, signs of drowsiness can be extracted.	Sudden changes in the head and eyes and changes in light intensity can decrease the percentage of drowsiness that is detected.
Based on behavioral and vehicle-based measurements (Hybrid methods)	In this method, infrared radiation is used for imaging, which allows imaging at night without disturbing the driver.	This method requires different categories in terms of image processing and status of eyes and face.

Monitoring the driver’s performance is a method that uses a camera to detect the driver’s drowsiness. Other ways to check the effects of different levels of drowsiness on the driver’s performance include studying the driver’s performance and the behavior of the vehicle by monitoring the angle of the steering wheel, acceleration and braking patterns, the vehicle’s speed, transverse momentum, and lateral shift. These methods are external techniques that are dependent on the circumstances and the type of vehicle (16). It seems that this method is the best way to detect drowsiness and lack of awareness, but it can be an annoying operation because of the need to install various sensors on the driver’s body. In contrast, detecting the rate at which the eyes blink is a precise method that is associated with minimal pressure and discomfort for drivers (17).

Studies have shown that when drowsiness occurs, the driver’s facial appearance undergoes significant changes, including changes in the eyes, head, mouth, and sitting position. Visual indicators of drowsiness can be observed using image processing techniques to assess the condition of the driver (12). Fatigue and drowsiness in individuals result in certain behaviors that can be observed easily by noting changes in the appearance of the eyes, head, and face. The most common visual characteristics of a drowsy person are longer blink rate duration, slow movement of the eyelids, eyelids seeming to close, closed eyes, frequent nodding of the head, yawning, fixed stare, and numbness. Machine vision is not an annoying technique to detect the visual signs of drowsy person (18). In this method, the pictures are taken by a camera placed in front of the driver, and, then, the desired signals are extracted by machine vision and image processing techniques. Most of the research that has been done on this group of methods involves examining the eyes and extracting signs of drowsiness from them. Examining the eyes is among the most successful methods for detecting drowsiness because it focuses special attention on changes in the eyes changes and their movements. The most important changes include a change in the blink rate, the duration of eye closures, and the direction in which the eyes are gazing (19).

Detection of drowsiness and eye tracking in a sequence of images has many applications, one of which is detecting a driver’s level of fatigue. Designing a system to detect a driver’s fatigue and sound an alarm in appropriate cases can prevent many accidents, many of which leads to the loss of life and property. Having information on the physiological state of a drowsy driver is the basic requirement for detecting drowsiness, and drivers must maintain their consciousness to find directions and guide the car (20). Some indicators to detect drowsiness through eye tracking are the three criteria of eye blink duration, eye blink frequency, and the percentage of eyelid closure (PERCLOS) over the pupil over time, reflecting slow eyelid closures, or “droops,” rather than blinks (17).

According to the results presented in table 1 and the proposed model in the figure 1, the result of work on driving simulator will be offered. The principle of this model is a new method based on a histogram of facial expressions (eyebrows, eyes, and mouth) that can be used to determine the status of the eyes, blinking, mouth, and eyebrows and to detect facial expressions. The proposed method seems to be more integrated than existing methods for the consideration of several factors of facial variables. It should be noted that, to date, methods of study have focused on one factor and most eye movements, and, actually, the problem with the previous studies was that the recognition software could not function properly if the people wore different eye glasses and there was lack of eye detection because it could not recognize drowsiness through other facial features. In addition, the proposed diagnostic model can be evaluated with objective variables in a driving simulator, including Standard Deviation of Lane Position (SDLP), Steering Wheel Movement (SWM), and subjective criteria, such as the driver’s self-reported measures of drowsiness on the Karolinsca Sleepiness Scale (KSS) and the Observer Rating of Drowsiness (ORD).

Figure 1.

Dynamic facial changes model for drowsiness detection

It seems that the most important approach for detecting a driver’s drowsiness is a combination of the various methods that have been used to detect drowsiness (17). In other words, among the existing methods, excluding non-physiological methods that interfere with driver and bother and annoyance her or him, a combination with other techniques should be used to detect drowsiness. Due to the high number of casualties in traffic accidents in Iran and other developing countries and the important role of drowsiness in severe accidents, then use of this hybrid model has the potential to reduce the high level of accidents caused by drowsiness and save the lives of many drivers and passengers.