Crowd Emotion and Behavior Analysis Using Lightweight CNN Model

International Journal of Electrical and Electronics Engineering

© 2024 by SSRG - IJEEE Journal

Volume 11 Issue 10

Year of Publication : 2024

Authors : Jignesh Vaniya, Safvan Vahora, Uttam Chauhan, Sudhir Vegad

10.14445/23488379/IJEEE-V11I10P104

How to Cite?

Jignesh Vaniya, Safvan Vahora, Uttam Chauhan, Sudhir Vegad, "Crowd Emotion and Behavior Analysis Using Lightweight CNN Model," SSRG International Journal of Electrical and Electronics Engineering, vol. 11,  no. 10, pp. 30-46, 2024. Crossref, https://doi.org/10.14445/23488379/IJEEE-V11I10P104

Abstract:

Crowd behavior is a critical aspect of numerous applications such as crowd management, urban planning, and safety monitoring in the current era of the world. Convolutional Neural Networks (CNNs), one of the most recent advancements in deep learning, have demonstrated potential in the analysis of crowd behavior patterns. However, computational limitations frequently make it difficult to implement complex CNN models for crowd analysis tasks, particularly in real-time applications. The utilization of a lightweight CNN model for crowd behavior analysis on the Motion Emotion Dataset (MED) is proposed in our study. The MED dataset has diverse scenes with varying crowd emotional and behavioral aspects, making it an ideal benchmark for evaluating crowd analysis algorithms. The 2D CNN model is applied to the MED datasets to extract the features and annotations for training the lightweight CNN. The model is validated in the validation set and achieved an accuracy of 99.4% on the Emotion Dataset and 94.35% on the Behavior Dataset. The results are validated using the confusion matrix. The results indicate that the lightweight CNN model achieves competitive performance on the MED dataset while exhibiting reduced computational overhead compared to more complex models. The discoveries made aid in the advancement of effective and scalable strategies for crowd surveillance and control, with applications spanning across diverse sectors such as public safety, transportation, and event coordination.

Keywords:

Crowd anomaly, Crowd behavior, CNN, Crowd emotional and behavioral analysis, Crowd Surveillance.

References:

[1] Swathi H.Y., G. Shivakumar, and H.S. Mohana, “Crowd Behavior Analysis: A Survey,” 2017 International Conference on Recent Ad-vances in Electronics and Communication Technology (ICRAECT), Bangalore, India, pp. 169-178, 2017.
[CrossRef] [Google Scholar] [Publisher Link]
[2] Barbara Krausz, and Christian Bauckhage, “Loveparade 2010: Automatic Video Analysis of a Crowd Disaster,” Computer Vision and Image Understanding, vol. 116, no. 3, pp. 307-319, 2012.
[CrossRef] [Google Scholar] [Publisher Link]
[3] Fatih Porikli et al., “Video Surveillance: Past, Present, and Now the Future [DSP Forum],” IEEE Signal Processing Magazine, vol. 30, no. 3, pp. 190-198, 2013.
[CrossRef] [Google Scholar] [Publisher Link]
[4] Mounir Bendali-Braham et al., “Recent Trends in Crowd Analysis: A Review,” Machine Learning with Applications, vol. 4, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[5] Bharath Varma Avs, List of Mob Lynching Incidents in India – 2019, Medium, 2019. [Online]. Available: https://medium.com/@bhara-thvarmaavs/list-of-mob-lynching-incidents-in-india-2019-5b97773f677f
[6] India’s Citizenship Protests - How over Three Months of Protests have Unfolded, Reuters Graphics, 2020. [Online]. Available: https://www.reuters.com/graphics/INDIA-CITIZENSHIP/PROTESTS/jxlbpgqlpqd/index.html
[7] Spriha Srivastava, India Revokes Special Status for Kashmir. Here’s what it Means, CNBC, 2019. [Online]. Available: https://www.cnbc.com/2019/08/05/article-370-what-is-happening-in-kashmir-india-revokes-special-status.html
[8] Prabhash K Dutta, Beyond JNU Violence: From Renaissance to Bloodshed, A Campus Story, India Today, 2020. [Online]. Available: https://www.indiatoday.in/news-analysis/story/beyond-jnu-when-university-campuses-different-jamia-amu-1634384-2020-01-06
[9] Amrit Dhillon, Students Protest across India after Attack at Top Delhi University, The Guardian, 2020. [Online]. Available: https://www.theguardian.com/world/2020/jan/06/students-injured-in-india-after-masked-attackers-raid-top-university
[10] Morgot Cohen, A History of Violence at Indian Universities, The National, 2010. [Online]. Available: https://www.thenational-news.com/world/asia/a-history-of-violence-at-indian-universities-1.557030
[11] Bhawana Tyagi, Swati Nigam, and Rajiv Singh, “A Review of Deep Learning Techniques for Crowd Behavior Analysis,” Archives of Computational Methods in Engineering, vol. 29, no. 7, pp. 5427-5455, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[12] India Protest: Farmers Breach Delhi's Red Fort in Huge Tractor Rally, BBC, 2021. [Online]. Available: https://www.bbc.com/news/uk-55793731
[13] Mehdi Moussaïd, Dirk Helbing, and Guy Theraulaz, “How Simple Rules Determine Pedestrian Behavior and Crowd Disasters,” Applied Physical Sciences, vol. 108, no. 17, pp. 6884-6888, 2011.
[CrossRef] [Google Scholar] [Publisher Link]
[14] Moin Nabi, Alessio Del Bue, and Vittorio Murino, “Temporal Poselets for Collective Activity Detection and Recognition,” 2013 IEEE International Conference on Computer Vision Workshops, Sydney, NSW, Australia, pp. 500-507, 2013.
[CrossRef] [Google Scholar] [Publisher Link]
[15] Barry G. Silverman et al., “Human Behavior Models for Agents in Simulators and Games: Part II: Gamebot Engineering with PMFserv,” Presence: Teleoperators and Virtual Environments, vol. 15, no. 2, pp. 163-185, 2006.
[CrossRef] [Google Scholar] [Publisher Link]
[16] Hamidreza Rabiee et al., “Emotion-Based Crowd Representation for Abnormality Detection,” arXiv, 2016.
[CrossRef] [Google Scholar] [Publisher Link]
[17] Noah Sulman et al., “How Effective is Human Video Surveillance Performance?,” 2008 19th International Conference on Pattern Recog-nition, Tampa, FL, USA, pp. 1-3, 2008.
[CrossRef] [Google Scholar] [Publisher Link]
[18] Gurjit Singh Walia, and Rajiv Kapoor, “Recent Advances on Multicue Object Tracking: A Survey,” Artificial Intelligence Review, vol. 46, no. 1, pp. 1-39, 2016.
[CrossRef] [Google Scholar] [Publisher Link]
[19] Jason M. Grant, and Patrick J. Flynn, “Crowd Scene Understanding from Video: A Survey,” ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), vol. 13, no. 2, pp. 1-23, 2017.
[CrossRef] [Google Scholar] [Publisher Link]
[20] Ovgu Ozturk, Toshihiko Yamasaki, and Kiyoharu Aizawa, “Detecting Dominant Motion Flows in Unstructured / Structured Crowd Scenes,” 2010 20th International Conference on Pattern Recognition, Istanbul, Turkey, pp. 3533-3536, 2010.
[CrossRef] [Google Scholar] [Publisher Link]
[21] Adriana Draghici, and Maarten Van Steen, “A Survey of Techniques for Automatically Sensing the Behavior of a Crowd,” ACM Compu-ting Surveys, vol. 51, no. 1, pp. 1-40, 2018.
[CrossRef] [Google Scholar] [Publisher Link]
[22] Smriti Srivastava et al., Concepts and Real-Time Applications of Deep Learning, 1st ed., Springer Cham, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[23] M. Sami Zitouni et al., “Advances and Trends in Visual Crowd Analysis: A Systematic Survey and Evaluation of Crowd Modelling Techniques,” Neurocomputing, vol. 186, pp. 139-159, 2016.
[CrossRef] [Google Scholar] [Publisher Link]
[24] Jie Zhang et al., “Collaboratively Self-Supervised Video Representation Learning for Action Recognition,” arXiv, 2024.
[CrossRef] [Google Scholar] [Publisher Link]
[25] H.Y. Swathi, and G. Shivakumar, “Hybrid Feature-Assisted Neural Model for Crowd Behavior Analysis,” SN Computer Science, vol. 2, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[26] Guodong Li, Lei Wang, and Minzhong Wu, “Crowd Behavior Intervention Based on Emotional Contagion,” Proceedings of the 2022 6th International Conference on Computer Science and Artificial Intelligence, pp. 1-6, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[27] Minzhong Wu, Lei Wang, and Guodong Li, “Crowd Emotion Recognition Based on Causal Spatiotemporal Structure,” Proceedings of the 8th International Conference on Computing and Artificial Intelligence (ICCAI '22), pp. 368-374, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[28] Ali Mollahosseini, Behzad Hasani, and Mohammad H. Mahoor, “Affectnet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild,” IEEE Transactions on Affective Computing, vol. 10, no. 1, pp. 18-31, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[29] Tian Wang et al., “Abnormal Event Detection via the Analysis of Multi-Frame Optical Flow Information,” Frontiers of Computer Science, vol. 14, pp. 304-313, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[30] Fernando J. Rendón-Segador et al., “Crimenet: Neural Structured Learning Using Vision Transformer for Violence Detection,” Neural Networks, vol. 161, pp. 318-329, 2023.
[CrossRef] [Google Scholar] [Publisher Link]
[31] Biao Guo et al., “Two-Stream Spatial-Temporal Auto-Encoder with Adversarial Training for Video Anomaly Detection,” IEEE Access, vol. 12, pp. 125881-125889, 2023.
[CrossRef] [Google Scholar] [Publisher Link]
[32] Manu Yadakere Murthygowda, Ravikumar Guralamata Krishnegowda, and Shashikala Salekoppalu Venkataramu, “An Integrated Multi-Level Feature Fusion Framework for Crowd Behaviour Prediction and Analysis,” Indonesian Journal of Electrical Engineering and Com-puter Science, vol. 30, no. 3, pp. 1369-1380, 2023.
[CrossRef] [Google Scholar] [Publisher Link]
[33] Nicolae-Catalin Ristea et al., “Self-Distilled Masked Auto-Encoders are Efficient Video Anomaly Detectors,” IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15984-15995, 2024.
[Google Scholar] [Publisher Link]
[34] Andra Acsintoae et al., “Ubnormal: New Benchmark for Supervised Open-Set Video Anomaly Detection,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20143-20153, 2022.
[Google Scholar] [Publisher Link]
[35] Guillermo del Castillo Torres et al., “Understanding How CNNs Recognize Facial Expressions: A Case Study with LIME and CEM,” Sensors, vol. 23, no. 1, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[36] Yu Tian et al., “Weakly-Supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Learning,” Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4975-4986, 2021.
[Google Scholar] [Publisher Link]
[37] Kian Yu Gan et al., “Contrastive-Regularized U-Net for Video Anomaly Detection,” IEEE Access, vol. 11, pp. 36658-36671, 2023.
[CrossRef] [Google Scholar] [Publisher Link]
[38] Alessandro Bruno et al., “High-Level Feature Extraction for Crowd Behaviour Analysis: A Computer Vision Approach,” Image Analysis and Processing. ICIAP 2022 Workshops, pp. 59-70, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[39] Siqi Wang et al., “Video Abnormal Event Detection by Learning to Complete Visual Cloze Tests,” arXiv Preprint, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[40] Elizabeth B. Varghese, and Sabu M. Thampi, “Towards the Cognitive and Psychological Perspectives of Crowd Behaviour: A Vision-Based Analysis,” Connection Science, vol. 33, no. 2, pp. 380-405, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[41] Xuguang Zhang et al., “Crowd Emotion Evaluation Based on Fuzzy Inference of Arousal and Valence,” Neurocomputing, vol. 445, pp. 194-205, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[42] Mark Marsden et al., “Resnetcrowd: A Residual Deep Learning Architecture for Crowd Counting, Violent Behaviour Detection and Crowd Density Level Classification,” 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Lecce, Italy, pp. 1-7, 2017.
[CrossRef] [Google Scholar] [Publisher Link]
[43] Khosro Rezaee et al., “A Survey on Deep Learning-Based Real-Time Crowd Anomaly Detection for Secure Distributed Video Surveil-lance,” Personal and Ubiquitous Computing, vol. 28, pp. 135-151, 2024.
[CrossRef] [Google Scholar] [Publisher Link]
[44] Fath U Min Ullah et al., “Violence Detection Using Spatiotemporal Features with 3D Convolutional Neural Network,” Sensors, vol. 19, no. 11, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[45] Introduction to Convolution Neural Network. [Online]. Available: https://www.geeksforgeeks.org/introduction-convolution-neural-net-work/
[46] Gaurav Tripathi, Kuldeep Singh, and Dinesh Kumar Vishwakarma, “Convolutional Neural Networks for Crowd Behaviour Analysis: A Survey,” The Visual Computer, vol. 35, pp. 753-776, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[47] Hamidreza Rabiee et al., “Novel Dataset for Fine-Grained Abnormal Behavior Understanding in Crowd,” 2016 13th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 95-101, Colorado Springs, CO, USA, 2016.
[CrossRef] [Google Scholar] [Publisher Link]
[48] Ven Jyn Kok, Mei Kuan Lim, and Chee Seng Chan, “Crowd Behavior Analysis: A Review Where Physics Meets Biology,” Neurocom-puting, vol. 177, pp. 342-362, 2016.
[CrossRef] [Google Scholar] [Publisher Link]
[49] A.A. Afiq et al., “A Review on Classifying Abnormal Behavior in Crowd Scene,” Journal of Visual Communication and Image Repre-sentation, vol. 58, pp. 285-303, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[50] Camille Dupont, Luis Tobías, and Bertrand Luvison, “Crowd-11: A Dataset for Fine Grained Crowd Behaviour Analysis,” 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA, pp. 9-16, 2017.
[CrossRef] [Google Scholar] [Publisher Link]
[51] Hamidreza Rabiee, Javad Haddadnia, and Hossein Mousavi, “Crowd Behavior Representation: An Attribute-Based Approach,” Spring-erPlus, vol. 5, no. 1, pp. 1-7, 2016.
[CrossRef] [Google Scholar] [Publisher Link]
[52] Lazaros Lazaridis, Anastasios Dimou, and Petros Daras, “Abnormal Behavior Detection in Crowded Scenes Using Density Heatmaps and Optical Flow,” 2018 26th European Signal Processing Conference (EUSIPCO), Rome, Italy, pp. 2060-2064, 2018.
[CrossRef] [Google Scholar] [Publisher Link]
[53] Elizabeth B. Varghese, and Sabu M. Thampi, “A Deep Learning Approach to Predict Crowd Behavior Based on Emotion,” Smart Multi-media, pp. 296-307, 2018.
[CrossRef] [Google Scholar] [Publisher Link]
[54] Elizabeth B. Varghese, Sabu M. Thampi, and Stefano Berretti, “A Psychologically Inspired Fuzzy Cognitive Deep Learning Framework to Predict Crowd Behavior,” IEEE Transactions on Affective Computing, vol. 13, no. 2, pp. 1005-1022, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[55] Motion Emotion Dataset (MED), Github. [Online]. Available: https://github.com/hosseinm/med