Let’s build and run an AI powered Drowsiness Detection Model
Application
According to NHTSA, approximately 1,500 people are killed and 71,000 are injured due to drowsy driving every year. Furthermore, around $12.5 Billion is lost every year because of crashes resulting from drowsy driving [1]. The German Road Safety Council (DVR) claims that one-in-four highway traffic fatalities are a result of momentary driver drowsiness [2]. These accidents are caused due to exertion, lack of sleep, hangovers and various other reasons. As such, it is evident that there is a need for a reliable driver drowsiness detection system that could alert a driver before a potential accident.
Dataset Collection
Before building any model, we need a sizable dataset so we went ahead and collected and used data from a variety of academic/open-source origins. For one, we used the UTA Real-Life Drowsiness Dataset curated by the University of Texas at Arlington . We also used data from the University of Technology Sydney (UTS) that involved data from 43 real-life participants [34]. However, while this data was helpful, it was not nearly enough to train our models to our desired accuracy levels. So we obtained OpenCV datasets from Data Flair as well as raw data collected and processed by the National Tsing Hua University (NTHU). The NTHU data consisted of data collected by recording multiple videos of at least 5 people (each diverse in ethnicity, shape, size, gender, clothing, facial hair, and location). This data contained light-skinned as well as dark-skinned faces, which was a unique facet to this dataset. Each set of videos in this dataset was recorded by a cell-phone camera or web camera with a resolution of 640 x 480 pixels at 15/30 frames per second (FPS).
We have also used a collection of online videos that were then converted into images. These were a collection of 30348 images obtained from researcher Jules-Raymond Tapamo at the University of Kwa-Zulu Natal [15]. These images were resized to 150 x 150 pixels before applying augmentation and feeding the data in batches to the model.
Below is a sample of the various images we used:
Figure 1 Dataset Collection from the University of Kwa-Zulu Natal [15]
Test Questionnaire and Test Site
Before the test, subjects were asked to complete a questionnaire on their basic information: age, gender, driving experience, nearsightedness, surgical history, and drug use. Those unqualified were excluded from this test.
For the NTHU data [38], videos and images were collected when the drivers were on National Highways. The test site consisted of controlled roads and testing roads. Controlled roads were around 2km long. The traffic flow during the data collection process was unobstructed. In order to eliminate the driver’s individual differences, the individual factors (gender, age, driving experience, level of education, etc) had to obey the uniform distribution.
Step 1 — Detecting the location of the Face
For the sake of simplicity, we are using a Haar-Cascade classifier model to detect faces and convert videos/images to grayscale color. The Python code below returns an array of detections with height and width of the bounding box of the object:
Here is a visualization of how the code will detect where the eyes are:
Figure 9.0 Face Positioning [24]
Step 2— Detecting the eyes
Next, we need to define the eye aspect ratio, which is used to compute the ratio of the distances between the vertical eye landmarks and the distances between the horizontal eye landmarks.
A function should be created to calculate eye aspect ratio:
The code above basically states that the return value of the eye aspect ratio will be approximately constant when the eye is open. The value will then become 0 once the model detects blinking.
If the eye is closed, the eye aspect ratio will return to constant, but the ratio will be much smaller than the ratio of the eye when it is open.
Figure 12.0 Eye Close Dimensions [27]
Step 3— Feeding into the AI model
Once the eye aspect ratio is calculated, we can then use that value along with other model components to classify a face as varying levels of drowsy in real-time.
A WAV/MP3 alarm should be triggered when the model detects drowsiness for over 5 seconds. [34]
Conclusion
Drowsy driving is a very real and prevalent problem in today’s society. As shown from data collected by NHTSA, drowsy driving claims nearly 1,500 lives each year while injuring 71,000 more. We not only compensate for this act through the loss of life, but also through over $12.5 Billion in tax-payer and insurance expenditure. What makes these statistics particularly disappointing is the fact that drowsy driving related deaths are easily preventable. With the widespread adoption of the appropriate Person-to-Vehicle (P2V) systems, the U.S could prevent almost all 1,500 fatalities claimed each year due to drowsy driving. Due to this fact, we prototyped a deep learning based ‘Drowsiness Detection’ model that could detect driver drowsiness. This model would be able to detect driver drowsiness in real-time and play a loud sound through the car’s speakers to alert the driver of their drowsy state.
After exploring a variety of different approaches in the form of SVM, SVM + CNN-HMM, and SVM + DBN, we determined that the SVM + DBN approach would render us with the most accurate results and the best overall model performance. While our SVM + DBN model had an overall accuracy of 82.08%, we believe there is still significant room for improvement through the use of emerging deep learning techniques such as Meta-Learning.
We firmly believe that our model, when implemented at scale, has the potential to reduce thousands of deaths attributable to drowsy driving every year.
Like our blog post? Send a message over on growthbotics!
Bibliography
[1] 2014. Dwivedi, Kartik. Biswaranjan, Kumar. Sethi, Amit. Drowsy driver detection using representation learning. India Institution of Technology. http://www.iitg.ac.in/amitsethi/publications/14.01DrowsyDeepNN.pdf
[2] 2019. Bamidele, Ameen Aliu. Kamardin, Kamilia. Non-intrusive Driver Drowsiness Detection based on Face and Eye Tracking. Razak School of Technology and Informatics. https://thesai.org/Downloads/Volume10No7/Paper_75-Non_Intrusive_Driver_Drowsiness_Detection.pdf
[3] 2019. Dataflair Team. Intermediate Python Project — Driver Drowsiness Detection System with OpenCV & Keras. Dataflair. https://data-flair.training/blogs/python-project-driver-drowsiness-detection-system/
[4] 2018. Tumen, Vedat. Yildirim, Ozal. Ergen, Burhan. Detection of driver drowsiness in driving environment using deep learning methods. https://ieeexplore.ieee.org/document/8391427
[5] 2019. Pearlmutter, Barak A. Hidden Markov Model Cheat Sheet. http://www.bcl.hamilton.ie/~barak/misc/hmm.pdf
[6] 2016. Rongrong, Fu. Wang, Hong. Zhao, Wenbo. Dynamic driver fatigue detection using hidden Markov model in real driving condition. https://www.sciencedirect.com/science/article/pii/S0957417416303293
[7] 2017. Deng, Chao. Wu, Chaozhong. Lyu, Nengchao. Huang, Zhen. Driving style recognition method using braking characteristics based on hidden Markov model. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0182419#pone.0182419.e016
[8] 2015. Berwick, R. An Idiot’s guide to Support vector machines (SVMs). MIT. http://web.mit.edu/6.034/wwwbob/svm-notes-long-08.pdf
[9] 2017. Deng, Chao. Wu, Chaozhong. Lyu, Nengchao. Huang, Zhen. Driving style recognition method using braking characteristics based on hidden Markov model. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0182419#pone.0182419.e016
[10] 2015. Deeplearning4j Tutorials. https://en.wikipedia.org/wiki/Deep_belief_network
[11] 2017. Missing Link AI. Deep Belief Networks: How They Work and What Are Their Applications. https://missinglink.ai/guides/neural-network-concepts/deep-belief-networks-work-applications/
[12] 2009. Hinton, Geoffrey. Deep Belief Networks. https://www.researchgate.net/publication/220580326_Deep_belief_networks
[13] 2014. Plis, Sergey. Deep learning for neuroimaging: A validation study. https://www.researchgate.net/publication/265393806_Deep_learning_for_neuroimaging_A_validation_study
[14] 2018. IceCream Labs. Deep Belief Networks — all you need to know. https://medium.com/@icecreamlabs/deep-belief-networks-all-you-need-to-know-68aa9a71cc53
[15] 2017. Chai, Rifai. Improving EEG-Based Driver Fatigue Classification Using Sparse-Deep Belief Networks. University of Sydney https://www.frontiersin.org/articles/10.3389/fnins.2017.00103/full
[16] 2016. Jadhav, Sarika. Drowsy Driver Detection using Representation Learning and Face Detection Technique. http://www.iitg.ac.in/amitsethi/publications/14.01DrowsyDeepNN.pdf
[17] 2017. Karim, Fazle. Majumdar, Somsjubra. Darabi, Houshang. LSTM Fully Convolutional Networks for Time Series Classification. https://github.com/kmutya/Algorithms-for-Drowsy-Driver-Detection
[18] 2019. Ngxande, Mkhuseli. Tapamo, Jule-Raymond. Burke, Michael. Detecting inter-sectional accuracy differences in driver drowsiness detection algorithms.
https://arxiv.org/pdf/1904.12631.pdfhttps://arxiv.org/pdf/1904.12631.pdf
[19] 2018. Morales, Cristyan Rufino. Drowsy Detection Using Facial Landmarks Extraction and Deep Neural Networks. https://dzone.com/articles/drowsy-detection-using-facial-landmarks-extraction
[20] 2014. Tadesse, Eyosiyas. Driver drowsiness detection through HMM based dynamic modeling.
[21] 2019. Gilbile, Prajakta. Driver’s Drowsiness Detection Using Image Processing.
[22] 2016. Rosa, Gustavo. Learning Parameters in Deep Belief Networks Through Firefly Algorithm.
[23] 2016. Chen, Chu-Song. Computer Vision — ACCV 2016 Workshops: ACCV 2016 International. https://books.google.com.tw/books?id=_9JcDgAAQBAJ&pg=PA130&lpg=PA130&dq=drowsy+detection+machine+learning+model+comparison&source=bl&ots=UYGkVHpL8G&sig=ACfU3U1VQbVIe45R1zfGVXGaI0ZjoNGvZw&hl=en&sa=X&ved=2ahUKEwjE--zgnaXmAhWHw4sBHe9jAts4FBDoATAGegQIChAB#v=onepage&q=drowsy&f=false
[24] 2017. Prasetio, Murman. Deep belief network optimization in speech recognition. https://ieeexplore.ieee.org/document/8304124
[25] 2016. Rosa, Gustavo. Papa, Joao. Costa, Kelton. Passos Junior, Leandro. Learning Parameters in Belief Networks Through Firefly Algorithm. https://www.researchgate.net/publication/305650436_Learning_Parameters_in_Deep_Belief_Networks_Through_Firefly_Algorithm
[26] 2006. Hinton, Geoffrey. Osindero, Simon. University of Toronto. A fast learning algorithm for deep belief nets. https://www.cs.toronto.edu/~hinton/absps/fastnc.pdf
[27] 2013. Sigari, Mohamad-Hoseyn. Fathy, Mahmood. Soryani, Mohsen. A Driver Face Monitoring System for Fatigue and Distraction Detection. https://www.hindawi.com/journals/ijvt/2013/263983/
[28] 2017. Karim, Fazle. Majumdar, Somsjubra. Darabi, Houshang. LSTM Fully Convolutional Networks for Time Series Classification. https://github.com/kmutya/Algorithms-for-Drowsy-Driver-Detection
[29] 2017. Mohit. Driver-Drowsiness-Detector. https://github.com/mohitwildbeast/Driver-Drowsiness-Detector
[30] 2019. Jaiswal, Ayush. MIT. Fatigue Detection System Based on Behavioral Characteristics of Driver. https://github.com/jaisayush/Fatigue-Detection-System-Based-On-Behavioural-Characteristics-Of-Driver
[31] 2016. Mero, Affro. Open CV Projects. https://github.com/affromero/OpenCV-Projects
[32] 2017. Bahadur, Akshay. Drowsiness Detection. https://github.com/akshaybahadur21/Drowsiness_Detection
[33] 2019. Kumar, Gautam. Development of Real-time Drowsiness Detection System using Python. https://medium.com/@gautamkumarjaiswal/development-of-real-time-drowsiness-detection-system-using-python-a87e3b2f58b6
[34] 2019. Vaghela, Jaynish. Driver Drowsiness Detection using LSTM Network.
[35] 2016. Guo, Quang. Wang, Fenglei. Lei, June. Tu, Dan. Li. Guohui. Convolutional feature learning and Hybrid CNN-HMM for scene number recognition. https://www.sciencedirect.com/science/article/abs/pii/S0925231215018950
[36] 2016. Guo, Quang. Wang, Fenglei. Lei, June. Tu, Dan. Li. Guohui. Convolutional feature learning and Hybrid CNN-HMM for scene number recognition.
https://www.sciencedirect.com/science/article/abs/pii/S0925231215018950
[37] 2019. Ngxande, Mkhuseli. Tapamo, Jule-Raymond. Burke, Michael. Detecting inter-sectional accuracy differences in driver drowsiness detection algorithms.
https://arxiv.org/pdf/1904.12631.pdf
[38] 2016. Weng, Ching-Hua. Lai, Ying-Hsiu. Lai, Shang-Hong. Driver Drowsiness Detection via a Hierarchical Temporal Deep Belief Network”. In Asian Conference on Computer Vision Workshop on Driver Drowsiness Detection from Video. http://cv.cs.nthu.edu.tw/php/callforpaper/datasets/DDD/
