Facial Emotion Recognition using a Modified Deep Convolutional Neural Network Based on the Concatenation of XCEPTION and RESNET50 V2

International Journal of Electrical and Electronics Engineering

© 2023 by SSRG - IJEEE Journal

Volume 10 Issue 6

Year of Publication : 2023

Authors : Sunil MP, Hariprasad .S.A

10.14445/23488379/IJEEE-V10I6P110

How to Cite?

Sunil MP, Hariprasad .S.A, "Facial Emotion Recognition using a Modified Deep Convolutional Neural Network Based on the Concatenation of XCEPTION and RESNET50 V2," SSRG International Journal of Electrical and Electronics Engineering, vol. 10,  no. 6, pp. 94-105, 2023. Crossref, https://doi.org/10.14445/23488379/IJEEE-V10I6P110

Abstract:

Facial emotion recognition has gained significant attention in modern years due to its wide applications in numerous fields, including human-computer interaction, market research and healthcare. This research focuses on improving facial emotion recognition accuracy by proposing a modified deep learning method based on the concatenation of Xception and ResNet50 architectures. The proposed approach aims to leverage the strengths of both Xception and ResNet50 networks to enhance facial expression representation and classification. Xception is known for its efficient feature extraction capabilities, while ResNet50 excels in capturing deeper and more complex patterns. By combining these architectures, the modified deep learning model can achieve higher emotion recognition accuracy. The research involves several stages. First, a large dataset of facial expressions is collected and preprocessed. The facial images are then fed into the modified deep-learning model, where feature extraction and classification occur. The model learns to recognize patterns and associations between facial expressions and specific emotions through a supervised learning process. Six distinct pre-trained DCNN models (ALEXNET, INCEPTIONV3, RESNET 50, VGG 16, XCEPTION and the concatenation of XCEPTION and RESNET50 V2) are used to validate the proposed system and with well-known datasets of FER2013, KDEF, CK+ JAFFE and with newly created custom Dataset-1 of 9K facial images. The proposed novel technique showed astounding accuracy, with a validation accuracy of 97.58% for a Softmax classifier, and it also recognized XCEPTION-RESNET V2 as the best network, with training and validation accuracy of 99.99% and 90%, respectively.

Keywords:

Deep learning, FER, Classifiers, Nets, Dataset.

References:

[1] E. Pranav et al., “Facial Emotion Recognition using Deep Convolutional Neural Network,” 2020 6th International Conference on Advanced Computing and Communication Systems, India, pp. 317-320, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[2] D. Y. Liliana, “Emotion Recognition from Facial Expression using Deep Convolutional Neural Networks,” Journal of Physics: Conference Series, vol. 1193, 2018.
[CrossRef] [Google Scholar] [Publisher Link]
[3] Salem Bin Saqer Al Marri, “Real Time Facial Emotion Recognition using Fast RCNN,” Thesis, Rochester Institute of Technology, United Arab Emirates, 2019.
[Google Scholar] [Publisher Link]
[4] Wfa Mellouk, and Wahida Handauzi, “Facial Emotion Recognition using Deep Learning: Review and Insights,” Procedia Computer Science, vol. 175, pp. 689-694, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[5] Zeynab Rzayeva, and Emin Alasgarov, “Facial Emotion Recognition Using Convolutional Neural Networks,” 2019 IEEE 13th International Conference on Application of Information and Communication Technologies, Azerbaijan, pp. 1-5, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[6] Xuan-Phung Hujuh, and Yong Gukkim, “Discrimination between Genuine and Fake Emotion using Long-Short Term Memory with Parametric Bias and Facial Landmarks,” IEEE International Conference on Computer Vision Workshop, pp. 3065-3062, 2017.
[Google Scholar] [Publisher Link]
[7] Min Seop Lee et al., “Emotion Recognition using a Convolutional Neural Network with Selected Statistical Photoplethysmogram Features,” Applied Sciences, vol. 10, no. 10, pp. 1-15, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[8] R. Santoshkumar, and M. Kalaiselvi Geetha, “Deep Learning Approach for Emotion Recognition from Human Body Movements with Feedforward Deep Convolutional Neural Networks,” Procedia Computer Science, vol. 152, pp. 158–165, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[9] Astha Sharma, “Emotion Recognition using Deep Convolutional Neural Networks with Large Scale Physiological Data,” Thesis, College of Engineering, University of South Florida, 2018.
[Google Scholar] [Publisher Link]
[10] S. Nithya Roopa, “Emotion Recognition from Facial Expressions using Deep Learning,” International Journal of Engineering and Advanced Technology, vol. 8, no. 6s, pp. 91-95, 2019.
[Google Scholar] [Publisher Link]
[11] Sowmiya R, Sivakamasundari G, and Archana V, “Facial Emotion Recognition using Deep Learning Approach,” 2022 International Conference on Automation, Computing and Renewable Systems, India, pp. 1064-1069, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[12] Sarwesh Giri et al., “Emotion Detection with Facial Feature Recognition using CNN & Open CV,” 2022 2nd International Conference on Advance Computing and Innovative Technologies in Engineering, India, pp. 230-232, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[13] Milind Rane et al., “Human Facial Emotion Recognition using Deep Learning Techniques,” 2023 Third International Conference on Advances in Electrical, Computing, Communication and Sustainable Technologies, India, pp. 1-6, 2023.
[CrossRef] [Google Scholar] [Publisher Link]
[14] Sabrina Begaj, Ali Osman Topal, and Maaruf Ali, “Emotion Recognition Based on Facial Expressions using Convolutional Neural Network (CNN),” 2020 International Conference on Computing, Networking, Telecommunications & Engineering Sciences Applications, Albania, pp. 58-63, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[15] Japleen Kaur et al., “Facial Emotion Recognition,” 2022 International Conference on Computational Intelligence and Sustainable Engineering Solutions, India, pp. 528-533, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[16] Tomoki Kusunose et al., “Facial Expression Emotion Recognition Based on Transfer Learning and Generative Model,” 2022 8th International Conference on Systems and Informatics, China, pp. 1-6, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[17] Tiehua Zhou et al., “Facial Expressions and Body Postures Emotion Recognition based on Convolutional Attention Network,” 2021 International Conference on Computer, Information and Telecommunication Systems, Turkey, pp. 1-5, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[18] Laura Martinez et al., “Contributions of Facial Expressions and Body Language to the Rapid Perception of Dynamic Emotions,” Cognition and Emotion, vol. 30, no. 5, pp. 939-952, 2016.
[CrossRef] [Google Scholar] [Publisher Link]
[19] Linkai Li et al., “Integrated Access Control System of Face Recognition and Non-Contact Temperature Measurement Based on Arduino,” International Journal of Computer and Organization Trends, vol. 12, no. 2, pp. 1-5, 2022.
[CrossRef] [Publisher Link]
[20] Xiaojie Li et al., “Recognizing Students' Emotions Based on Facial Expression Analysis,” 2021 11th International Conference on Information Technology in Medicine and Education, China, pp. 96-100, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[21] Ankit Jain et al., “An Overview on Facial Expression Perception Mechanisms,” SSRG International Journal of Computer Science and Engineering, vol. 6, no. 4, pp. 19-24, 2019.
[CrossRef] [Publisher Link]
[22] Shrey Modi, and Mohammed Husain Bohara, “Facial Emotion Recognition using Convolution Neural Network,” 2021 5th International Conference on Intelligent Computing and Control Systems, India, pp. 1339-1344, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[23] Özay Ezerceli, and M. Taner Eskil, “Convolutional Neural Network (CNN) Algorithm Based Facial Emotion Recognition (FER) System for FER-2013 Dataset,” 2022 International Conference on Electrical, Computer, Communications and Mechatronics Engineering, Maldives, pp. 1-6, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[24] V. Pandimurugan, Angad Singh, and Akash Tiwari, “Facial Emotion Recognition for Students Using Machine Learning,” 2023 International Conference on Computer Communication and Informatics, India, pp. 1-4, 2023.
[CrossRef] [Google Scholar] [Publisher Link]
[25] Pooja G. Nair, and Sneha R, “A Review: Facial Recognition using Machine Learning,” International Journal of Recent Engineering Science, vol. 7, no. 3, pp. 85-89, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[26] Mehmet Akif Ozdemir et al., “Deep Learning Based Facial Emotion Recognition System,” 2020 Medical Technologies Congress, Turkey, pp. 1-4, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[27] Nikhil Kumar Singh, and Gokul Rajan V, “Facial Emotion Recognition in Python,” SSRG International Journal of Computer Science and Engineering, vol. 7, no. 6, pp. 20-23, 2020.
[CrossRef] [Publisher Link]
[28] Faza N. Azizi, Arrie Kurniawardhani, and Irving V. Paputungan, “Facial Expression Image based Emotion Detection using Convolutional Neural Network,” 2022 IEEE 20th Student Conference on Research and Development, Malaysia, pp. 157-162, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[29] Zeng Runhua, and Zhang Shuqun, “Improving Speech Emotion Recognition Method of Convolutional Neural Network,” International Journal of Recent Engineering Science, vol. 5, no. 3, pp. 1-7, 2018.
[CrossRef] [Google Scholar] [Publisher Link]
[30] Ishika Agrawal et al., “Emotion Recognition from Facial Expression using CNN,” 2021 IEEE 9th Region 10 Humanitarian Technology Conference, India, pp. 01-06, 2021.
[CrossRef] [Google Scholar] [Publisher Link]