Adaptive IoT Botnet Defense: Combining Hybrid Deep Learning and Real-Time SDN Mitigation

International Journal of Computer Science and Engineering

© 2025 by SSRG - IJCSE Journal

Volume 12 Issue 3

Year of Publication : 2025

Authors : Preeti Kailas Suryawanshi, Sonal Kirankumar Jagtap

10.14445/23488387/IJCSE-V12I3P105

How to Cite?

Preeti Kailas Suryawanshi, Sonal Kirankumar Jagtap, "Adaptive IoT Botnet Defense: Combining Hybrid Deep Learning and Real-Time SDN Mitigation," SSRG International Journal of Computer Science and Engineering , vol. 12,  no. 3, pp. 33-39, 2025. Crossref, https://doi.org/10.14445/23488387/IJCSE-V12I3P105

Abstract:

The rapid expansion of the Internet of Things (IoT) has led to botnet attacks, which use compromised devices for malicious activities such as Distributed Denial-of-Service (DDoS) attacks and data breaches. Traditional rule-based intrusion detection systems struggle to detect these new threats, which demand advanced machine learning (ML) and deep learning (DL) models. In this paper, a hybrid CNN-RNN model employing both spatial and temporal analysis of traffic is proposed for better IoT botnet detection. Federated learning also maintains privacy during model training, and Graph Neural Networks (GNNs) improve botnet behavior modeling. A Software-Defined Networking (SDN)-based mitigation method is employed for providing real-time response with rapid isolation of malicious traffic. To counter IoT resource constraints, model optimization techniques such as pruning and quantization are employed. Experimental evaluations using the UNSW-NB15 dataset demonstrate superior detection accuracy (99.1), with minimal false positives over traditional approaches. These findings recognize the potential of hybrid deep learning and SDN-based solutions for effective, real-time IoT botnet protection.

Keywords:

IoT security, Machine learning, Hybrid IDS, Anomaly detection, Botnet detection, Intrusion detection systems.

References:

[1] Ayush Kumar et al., “Machine Learning-Based Early Detection of IoT Botnets Using Network-Edge Traffic,” Computers & Security, vol. 117, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[2] Jeeyung Kim et al., “Botnet Detection Using Recurrent Variational Autoencoder,” IEEE Global Communications Conference, Taipei, Taiwan, pp. 1-6, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[3] Jiawei Zhou et al., “Automating Botnet Detection with Graph Neural Networks,” arXiv, pp. 1-8, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[4] Nelly Elsayed, Zag ElSayed, and Magdy Bayoumi, “IoT Botnet Detection Using an Economic Deep Learning Model,” IEEE World AI IoT Congress, Seattle, WA, USA, pp. 134-142, 2023. [CrossRef] [Google Scholar] [Publisher Link]
[5] A. Karthick Kumar et al., “Enhanced Hybrid Deep Learning Approach for Botnet Attacks Detection in IoT Environment,” 7th International Conference on Signal Processing and Information Security, Dubai, United Arab Emirates, pp. 1-6, 2024.
[CrossRef] [Google Scholar] [Publisher Link]
[6] Shamsul Haq, and Yashwant Singh, “Botnet Detection using Machine Learning,” 2018 Fifth International Conference on Parallel, Distributed and Grid Computing (PDGC), Solan, India, pp. 240-245, 2018.
[CrossRef] [Google Scholar] [Publisher Link]
[7] MohammadNoor Injadat, Abdallah Moubayed, and Abdallah Shami, “Detecting Botnet Attacks in IoT Environments: An Optimized Machine Learning Approach,” 32nd International Conference on Microelectronics, Aqaba, Jordan, pp. 1-4, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[8] Yair Meidan et al., “N-BaIoT: Network-Based Detection of IoT Botnet Attacks Using Deep Autoencoders,” IEEE Pervasive Computing, vol. 17, no. 3, pp. 12-22, 2018. [CrossRef] [Google Scholar] [Publisher Link]
[9] S. García et al., “An Empirical Comparison of Botnet Detection Methods,” Computers & Security, vol. 45, pp. 100-123, 2014.
[CrossRef] [Google Scholar] [Publisher Link]
[10] Mauro Conti et al., “Internet of Things Security and Forensics: Challenges and Opportunities,” Future Generation Computer Systems, vol. 78, no. 2, pp. 544-546, 2018.
[CrossRef] [Google Scholar] [Publisher Link]
[11] Nikola Milosevic, Ali Dehghantanha, and Kim-Kwang Raymond Choo, “Machine Learning Aided Android Malware Classification,” Computers & Electrical Engineering, vol. 61, pp. 266-274, 2017.
[CrossRef] [Google Scholar] [Publisher Link]
[12] Abbas Yazdinejad et al., “Cryptocurrency Malware Hunting: A Deep Recurrent Neural Network Approach,” Applied Soft Computing, vol. 96, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[13] Mahsa Nazemi Gelian, Hoda Mashayekhi, and Yoosof Mashayekhi, “A Self-Learning Stream Classifier for Flow-Based Botnet Detection,” International Journal of Communication Systems, vol. 32, no. 16, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[14] Arvind Prasad, and Shalini Chandra, “Machine Learning to Combat Cyberattack: A Survey of Datasets and Challenges,” Journal of Defense Modeling & Simulation, vol. 20, no. 4, pp. 577-588, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[15] Mohammed Mudassir et al., “Detection of Botnet Attacks against Industrial IoT Systems by Multilayer Deep Learning Approaches,” Wireless Communications and Mobile Computing, vol. 2022, no. 1, pp. 1-12, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[16] P. Panimalar, and K. Rameshkumar, “A Novel Traffic Analysis Model for Botnet Discovery in Dynamic Network,” Arabian Journal for Science and Engineering, vol. 44, no. 4, pp. 3033-3042, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[17] Majda Wazzan et al., “Internet of Things Botnet Detection Approaches: Analysis and Recommendations for Future Research,” Applied Sciences, vol. 11, no. 12, pp. 1-46, 2021.
[CrossRef] [Google Scholar] [Publisher Link] 
[18] Gulbadan Khehra, and Sanjeev Sofat, “Botnet Detection Techniques: A Review,” Second International Conference on Intelligent Computing and Control Systems, Madurai, India, pp. 1319-1326, 2018.
[CrossRef] [Google Scholar] [Publisher Link]
[19] Sahar Aldhaheri et al., “Artificial Immune Systems Approaches to Secure the Internet of Things: A Systematic Review of the Literature and Recommendations for Future Research,” Journal of Network and Computer Applications, vol. 157, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[20] Donghui Hu et al., “A Blockchain-Based Trading System for Big Data,” Computer Networks, vol. 191, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[21] Shweta Saharan et al., “Scaling & Fuzzing: Personal Image Privacy from Automated Attacks in Mobile Cloud Computing,” Journal of Information Security and Applications, vol. 60, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[22] A. Anish Halimaa, and K. Sundarakantham, “Machine Learning Based Intrusion Detection Systems,” 2019 3rd International Conference on Trends in Electronics and Informatics (ICOEI), Tirunelveli, India, pp. 916-920, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[23] Jia Wei et al., “DPLRS: Distributed Population Learning Rate Schedule,” Future Generation Computer Systems, vol. 132, pp. 40-50, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[24] Alaa Tolah, Steven M. Furnell, and Maria Papadaki, “An Empirical Analysis of the Information Security Culture Key Factors Framework,” Computers & Security, vol. 108, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[25] Shigenari Nakamura, Tomoya Enokido, and Makoto Takizawa, “Information Flow Control Based on Capability Token Validity for Secure IoT: Implementation and Evaluation,” Internet of Things, vol. 15, 2021.
[CrossRef] [Google Scholar] [Publisher Link
[26] Satish Pokhrel, Robert Abbas, and Bhulok Aryal “IoT Security: Botnet Detection in IoT using Machine Learning,” arXiv, pp. 1-11, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[27] Amirfarhad Nilizadeh et al., “Adaptive Matrix Pattern Steganography on RGB Images,” Journal of Cyber Security and Mobility, vol. 11, no. 1, pp. 1-28, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[28] Abdurrahman Pektaş, and Tankut Acarman, “Botnet Detection based on Network Flow Summary and Deep Learning,” International Journal of Network Management, vol. 28, no. 6, 2018.
[CrossRef] [Google Scholar] [Publisher Link]
[29] Swapnil Dhamal et al., “Strategic Investments in Distributed Computing: A Stochastic Game Perspective,” Journal of Parallel and Distributed Computing, vol. 169, pp. 317-333, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[30] Weiping Zhang et al., “Variational Learning of Deep Fuzzy Theoretic Nonparametric Model,” Neurocomputing, vol. 506, pp. 128-145, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[31] Francesc Wilhelmi, Lorenza Giupponi, and Paolo Dini, “Analysis and Evaluation of Synchronous and Asynchronous FLchain,” Computer Networks, vol. 218, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[32] Segun I. Popoola et al., “Hybrid Deep Learning for Botnet Attack Detection in the Internet-of-Things Networks,” IEEE Internet of Things Journal, vol. 8, no. 6, pp. 4944-4956, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[33] Ziheng Wang et al., “LogSC: Model-Based One-Sided Communication Performance Estimation,” Future Generation Computer Systems, vol. 132, pp. 25-39, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[34] Qasem Abu Al-Haija, and Mu’awya Al-Dala’ien, “ELBA-IoT: An Ensemble Learning Model for Botnet Attack Detection in IoT Networks,” Journal of Sensor and Actuator Networks, vol. 11, no. 1, pp. 1-15, 2022.
[CrossRef] [Google Scholar] [Publisher Link]