UNet and Transformer-Based Model for Multi-Modality Brain Tumor Segmentation

International Journal of Electronics and Communication Engineering

© 2023 by SSRG - IJECE Journal

Volume 10 Issue 8

Year of Publication : 2023

Authors : Jayashree Shedbalkar, K. Prabhushetty, R. H. Havaldar

10.14445/23488549/IJECE-V10I8P103

How to Cite?

Jayashree Shedbalkar, K. Prabhushetty, R. H. Havaldar, "UNet and Transformer-Based Model for Multi-Modality Brain Tumor Segmentation," SSRG International Journal of Electronics and Communication Engineering, vol. 10,  no. 8, pp. 22-35, 2023. Crossref, https://doi.org/10.14445/23488549/IJECE-V10I8P103

Abstract:

Currently, the human race is facing several health-related issues where brain tumours are recognized as one of the leading causes of morbidity and mortality worldwide. Several researches and surveys have reported that the timely detection and prediction of brain tumours can help prevent their diverse impacts. Therefore, segmentation of these brain tumors is one of the prime tasks to proliferate the accuracy of diagnosis. Deep learning has become a promising technique to facilitate an automated Brain Tumor Segmentation (BTS) approach. Most current deep learning approaches rely on Convolutional Neural Networks (CNNs), which fail to contain long-term dependencies and global context information. Moreover, the performance of these systems is affected due to receptive field limitations during convolution operations. Currently, UNet-based architectures are adopted to perform medical image segmentation. Thus, in this work, we considered UNet as the base model for segmentation and incorporated transformer-based modules to improvise the segmentation accuracy; along with this, we present a hybrid attention mechanism that uses local and global context information. Based on this architecture, we evaluated the efficiency of the proposed approach for various Brats Datasets (2015, 2017, 2019, 2020 and 2021). The proposed approach achieves the average dice score of 0.94, 0.921, 0.83, and 0.94 for Brat's dataset.

Keywords:

Unet architecture, Brain tumor segmentation, Transformer based module, Deep learning model, Swin transform.

References:

[1] Rebecca L. Siegel, Kimberly D. Miller, and Ahmedin Jemal, “Cancer Statistics, 2015,” A Cancer Journal for Clinicians, vol. 65, no. 1, pp. 5-29, 2015.
[CrossRef] [Google Scholar] [Publisher Link]
[2] Rebecca L. Siegel, Kimberly D. Miller, and Ahmedin Jemal, “Cancer Statistics,2017,” A Cancer Journal for Clinicians, vol. 67, no. 1, pp. 7-30, 2017.
[CrossRef] [Google Scholar] [Publisher Link]
[3] Gopal S. Tandel et al., “A Review on a Deep Learning Perspective in Brain Cancer Classification,” Cancers, vol. 11, no. 1, pp. 1-32, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[4] Amin Zadeh Shirazi et al., “The Application of Deep Convolutional Neural Networks to Brain Cancer Images: A Survey,” Journal of Personalized Medicine, vol. 10, no. 4, pp. 1-27, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[5] Mahwash Mukhtar et al., “Nanomaterials for Diagnosis and Treatment of Brain Cancer: Recent Updates,” Chemosensors, vol. 8, no. 4, pp. 1-31, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[6] Asaf Raza et al., “A Hybrid Deep Learning-Based Approach for Brain Tumor Classification,” Electronics, vol. 11, no. 7, pp. 1-17, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[7] Mohammed Rasool et al., “A Hybrid Deep Learning Model for Brain Tumour Classification,” Entropy, vol. 24, no. 6, pp. 1-16, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[8] Neelum Noreen et al., “A Deep Learning Model Based on Concatenation Approach for the Diagnosis of Brain Tumor,” IEEE Access, vol. 8, pp. 55135-55144, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[9] Rajan Hossain, Roliana Binti Ibrahim, and Haslina Binti Hashim, “Automated Brain Tumor Detection using Machine Learning: A Bibliometric Review,” World Neurosurgery, vol. 175, pp. 57-68, 2023.
[CrossRef] [Google Scholar] [Publisher Link]
[10] Philipp Lohmann et al., “Combined Amino Acid Positron Emission Tomography and Advanced Magnetic Resonance Imaging in Glioma Patients,” Cancers, vol. 11, no. 2, pp. 1-13, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[11] H. J. Biersack, F. Grünwald, and J. Kropp, “Single Photon Emission Computed Tomography Imaging of Brain Tumors,” Seminars in Nuclear Medicine, vol. 21, no. 1, pp. 2-10, 1991.
[CrossRef] [Google Scholar] [Publisher Link]
[12] Jonathan Huang et al., “Artificial Intelligence Applications in Pediatric Brain Tumor Imaging: A Systematic Review,” World Neurosurgery, vol. 157, pp. 99-105, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[13] Muhammad Farhan Safdar, Shayma Saad Alkobaisi, and Fatima Tuz Zahra, “A Comparative Analysis of Data Augmentation Approaches for Magnetic Resonance Imaging (MRI) Scan Images of Brain Tumor,” Acta informatica medica, vol. 28, no. 1, pp. 29-36, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[14] Adrien Heintz, and Jean-Marc Constans, “Role of Magnetic Resonance Spectroscopy in Clinical Management of Brain Tumors,” Atlas of Clinical Cases on Brain Tumor Imaging, pp. 49-67, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[15] R. Sakthi Prabha, and M. Vadivel, “Brain Tumor Stages Prediction using FMS-DLNN Classifier and Automatic RPO-RG Segmentation,” SSRG International Journal of Electrical and Electronics Engineering, vol. 10, no. 2, pp. 110-121, 2023.
[CrossRef] [Publisher Link]
[16] Javaria Amin et al., “Brain Tumor Detection using Statistical and Machine Learning Method,” Computer Methods and Programs in Biomedicine, vol. 177, pp. 69-79, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[17] Biswajit Jena, Gopal Krishna Nayak, and Sanjay Saxena, “An Empirical Study of Different Machine Learning Techniques for Brain Tumor Classification and Subsequent Segmentation using Hybrid Texture Feature,” Machine Vision and Applications, vol. 33, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[18] Dayong Wang et al., “Deep Learning for Identifying Metastatic Breast Cancer,” arXiv:1606.05718, 2016.
[CrossRef] [Google Scholar] [Publisher Link]
[19] Wejdan L. Alyoubi, Wafaa M. Shalash, and Maysoon F. Abulkhair, “Diabetic Retinopathy Detection through Deep Learning Techniques: A Review,” Informatics in Medicine Unlocked, vol. 20, p. 100377, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[20] Zhihui Guo et al., “Deep LOGISMOS: Deep Learning Graph-Based 3D Segmentation of Pancreatic Tumors on CT Scans,” 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), Washington, DC, USA, pp. 1230-1233, 2018.
[CrossRef] [Google Scholar] [Publisher Link]
[21] R. Tamilaruvi et al., “Brain Tumor Detection in MRI Images using Convolutional Neural Network Technique,” SSRG International Journal of Electrical and Electronics Engineering, vol. 9, no. 12, pp. 198-208, 2022.
[CrossRef] [Publisher Link]
[22] Jonathan Long, Evan Shelhamer, and Trevor Darrell, “Fully Convolutional Networks for Semantic Segmentation,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431-3440. 2015.
[Google Scholar] [Publisher Link]
[23] Olaf Ronneberger, Philipp Fischer, and Thomas Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015, Munich, Germany, pp. 234-241, 2015.
[CrossRef] [Google Scholar] [Publisher Link]
[24] Zongwei Zhou et al., “UNet++: A Nested U-Net Architecture for Medical Image Segmentation,” ML-CDS 2018: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Granada, Spain, pp. 3-11, 2018.
[CrossRef] [Google Scholar] [Publisher Link]
[25] Ahmed Shaker, Wai Yeung Yan, and Paul E. LaRocque, “Automatic Land-Water Classification using Multispectral Airborne LiDAR Data for Near-Shore and River Environments,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 152, pp. 94-108, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[26] D. Napoleon, and M. Praneesh, “Detection of Brain Tumor using Kernel Induced Possiblistic C-Means Clustering,” International Journal of Computer Organization Trends and Technology (IJCOT), vol. 3, no. 5, pp. 40-42, 2013.
[Google Scholar] [Publisher Link]
[27] Xiaolong Wang et al., “Non-Local Neural Networks,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794-7803, 2018.
[Google Scholar] [Publisher Link]
[28] Jo Schlemper et al., “Attention Gated Networks: Learning to Leverage Salient Regions in Medical Images,” Medical Image Analysis, vol. 53, pp. 197-207, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[29] Ashish Vaswani et al., “Attention is All you Need,” 31st Conference on Neural Information Processing Systems (NIPS), Long Beach, CA, USA, pp. 1-11, 2017.
[Google Scholar] [Publisher Link]
[30] Umit Ilhan, and Ahmet Ilhan, “Brain Tumor Segmentation Based on a New Threshold Approach,” Procedia Computer Science, vol. 120, pp. 580-587, 2017.
[CrossRef] [Google Scholar] [Publisher Link]
[31] Jakhongir Nodirov, Akmalbek Bobomirzaevich Abdusalomov, and Taegkeun Whangbo, “Attention 3D U-Net with Multiple Skip Connections for Segmentation of Brain Tumor Images,” Sensors, vol. 22, no. 17, pp. 1-17, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[32] Venkatesh, and M. Judith Leo, “MRI Brain Image Segmentation and Detection using K-NN Classification,” Journal of Physics: Conference Series, vol. 1362, no. 1, p. 012073, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[33] Baoshi Chen et al., “A Novel Extended Kalman Filter with Support Vector Machine Based Method for the Automatic Diagnosis and Segmentation of Brain Tumors,” Computer Methods and Programs in Biomedicine, vol. 200, p. 105797, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[34] Rohan K. Gajre, Savita A. Lothe, and Santosh G. Vishwakarma, “Identification of Brain Tumor using Image Processing Technique: Overviews of Methods,” SSRG International Journal of Computer Science and Engineering, vol. 3, no. 10, pp. 48-52, 2016.
[CrossRef] [Google Scholar] [Publisher Link]
[35] Michał Futrega et al., “Optimized U-Net for Brain Tumor Segmentation,” BrainLes 2021: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, pp. 15-29, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[36] Yun Jiang et al., “SwinBTS: A Method for 3D Multimodal Brain Tumor Segmentation using Swin Transformer,” Brain Sciences, vol. 12, no. 6, pp. 1-15, 2022. [CrossRef] [Google Scholar] [Publisher Link]
[37] Yujian Liu et al., “Scale-Adaptive Super-Feature Based MetricUNet for Brain Tumor Segmentation,” Biomedical Signal Processing and Control, vol. 73, p. 103442, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[38] Necip Cinar, Alper Ozcan, and Mehmet Kaya, “A Hybrid DenseNet121-UNet Model for Brain Tumor Segmentation from MR Images,” Biomedical Signal Processing and Control, vol. 76, p. 103647, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[39] Junying Zeng et al., “Improved U-Net3+ with Stage Residual for Brain Tumor Segmentation,” BMC Medical Imaging, vol. 22, pp. 1-15, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[40] D. Gai et al., “RMTF-Net: Residual Mix Transformer Fusion Net for 2D Brain Tumor Segmentation,” Brain Sciences, vol. 12, no. 9, pp. 1-18, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[41] Junjie Liang et al., “TransConver: Transformer and Convolution Parallel Network for Developing Automatic Brain Tumor Segmentation in MRI Images,” Quantitative Imaging in Medicine and Surgery, vol. 12, no. 4, pp. 2397-2415, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[42] Spyridon Bakas, MICCAI BRATS - The Multimodal Brain Tumor Segmentation Challenge, 2021. [Online]. Available: http://braintumorsegmentation.org/
[43] Li ZongRen et al., “DenseTrans: Multimodal Brain Tumor Segmentation using Swin Transformer,” IEEE Access, vol. 11, pp. 42895- 42908, 2023.
[CrossRef] [Google Scholar] [Publisher Link]
[44] Yu Chen et al., “CSU-Net: A CNN-Transformer Parallel Network for Multimodal Brain Tumour Segmentation,” Electronics, vol. 11, no. 14, pp. 1-12, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[45] Dhiraj Maji, Prarthana Sigedar, and Munendra Singh, “Attention Res-UNet with Guided Decoder for Semantic Segmentation of Brain Tumors,” Biomedical Signal Processing and Control, vol. 71, p. 103077, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[46] Ze Liu et al., “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows,” Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022. 2021.
[Google Scholar] [Publisher Link]
[47] Kuan-Lun Tseng et al., “Joint Sequence Learning and Cross-Modality Convolution for 3D Biomedical Segmentation,” Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 6393-6400. 2017.
[Google Scholar] [Publisher Link]
[48] Loic Le Folgoc et al., “Lifted Auto-Context Forests for Brain Tumour Segmentation,” BrainLes 2016: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, pp. 171-183, 2017.
[CrossRef] [Google Scholar] [Publisher Link]
[49] Edgar A. Rios Piedra et al., “Brain Tumor Segmentation by Variability Characterization of Tumor Boundaries,” BrainLes 2016: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, pp. 206-216, 2016.
[CrossRef] [Google Scholar] [Publisher Link]
[50] Bi Song et al., “Anatomy-Guided Brain Tumor Segmentation and Classification,” BrainLes 2016: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, Berlin/Heidelberg, Germany, pp. 162-170, 2016.
[CrossRef] [Google Scholar] [Publisher Link]
[51] Jiten Chaudhary, Rajneesh Rani, and Aman Kamboj, “Deep Learning-Based Approach for Segmentation of Glioma Sub-Regions in MRI,” International Journal of Intelligent Computing and Cybernetics, vol. 13, no. 4, pp. 389-406, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[52] Saddam Hussain, Syed Muhammad Anwar, and Muhammad Majid, “Segmentation of Glioma Tumors in Brain using Deep Convolutional Neural Network, Neurocomputing, vol. 282, pp. 248-261, 2018.
[CrossRef] [Google Scholar] [Publisher Link]
[53] Mahmoud Elmezain et al., “Brain Tumor Segmentation using Deep Capsule Network and Latent-Dynamic Conditional Random Fields,” Journal of Imaging, vol. 8, no. 7, pp. 1-16, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[54] Lele Chen et al., “MRI Tumor Segmentation with Densely Connected 3D CNN,” Medical Imaging 2018: Image Processing, Houston, Texas, US, vol. 10574, 2018.
[CrossRef] [Google Scholar] [Publisher Link]
[55] Konstantinos Kamnitsas et al., “Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation,” Medical Image Analysis, vol. 36, pp. 61-78, 2017.
[CrossRef] [Google Scholar] [Publisher Link]
[56] Hao Dong et al., “Automatic Brain Tumor Detection and Segmentation using U-Net Based Fully Convolutional Networks,” MIUA 2017: Medical Image Understanding and Analysis, pp. 506-517, 2017.
[CrossRef] [Google Scholar] [Publisher Link]
[57] Sérgio Pereira et al., “Brain Tumor Segmentation using Convolutional Neural Networks in MRI Images,” IEEE Transactions on Medical Imaging, vol. 35, no. 5, pp. 1240-1251, 2016.
[CrossRef] [Google Scholar] [Publisher Link]
[58] Adel Kermi, Issam Mahmoudi, and Mohamed Tarek Khadir, “Deep Convolutional Neural Networks using U-Net for Automatic Brain Tumor Segmentation in Multimodal MRI,” BrainLes 2018: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, pp. 37-48, 2018.
[CrossRef] [Google Scholar] [Publisher Link]
[59] Xiaomei Zhao et al., “A Deep Learning Model Integrating FCNNs and CRFs for Brain Tumor Segmentation,” Medical Image Analysis, vol. 43, pp. 98-111, 2018.
[CrossRef] [Google Scholar] [Publisher Link]
[60] Xiangyu Zhao et al., “Prior Attention Network for Multi-Lesion Segmentation in Medical Images,” IEEE Transactions on Medical Imaging, vol. 41, no. 12, pp. 3812-3823, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[61] Mina Ghaffari et al., “Automated Post-Operative Brain Tumour Segmentation: A Deep Learning Model Based on Transfer Learning from Pre-Operative Images,” Magnetic Resonance Imaging, vol. 86, pp. 28-36, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[62] Xi Guan et al., “3D AGSE-VNet: An Automatic Brain Tumor MRI Data Segmentation Framework,” BMC Medical Imaging, vol. 22, no. 1, pp. 1-18, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[63] Yujian Liu et al., “Scale-Adaptive Super-Feature Based MetricUNet for Brain Tumor Segmentation,” Biomedical Signal Processing and Control, vol. 73, p. 103442, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[64] Jindong Sun et al., “Segmentation of the Multimodal Brain Tumor Image Used the Multi-Pathway Architecture Method Based on 3D FCN,” Neurocomputing, vol. 423, pp. 34-45, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[65] Ning Sheng et al., “Second-Order ResU-Net for Automatic MRI Brain Tumor Segmentation,” Mathematical Biosciences and Engineering, vol. 18, no. 5, pp. 4943-4960, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[66] Wenxuan Wang et al., “TransBTS: Multimodal Brain Tumor Segmentation using Transformer,” Medical Image Computing and Computer Assisted Intervention-MICCAI 2021, Strasbourg, France, pp. 109-119, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[67] Pengyu Li et al., “Automatic Brain Tumor Segmentation from Multiparametric MRI Based on Cascaded 3D U-Net and 3D U-Net++,” Biomedical Signal Processing and Control, vol. 78, p. 103979, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[68] Xiangbin Liu et al., “Attention-Based Multimodal Glioma Segmentation with Multi-Attention Layers for Small-Intensity Dissimilarity,” Journal of King Saud University - Computer and Information Sciences, vol. 35, no. 4, pp. 183-195, 2023.
[CrossRef] [Google Scholar] [Publisher Link]
[69] Théophraste Henry et al., “Brain Tumor Segmentation with Self-Ensembled, Deeply-Supervised 3D U-Net Neural Networks: A BraTS 2020 Challenge Solution,” BrainLes 2020: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, Lima, Peru, pp. 327-339, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[70] Navchetan Awasthi, Rohit Pardasani, and Swati Gupta, “Multi-Threshold Attention U-Net (MTAU) Based Model for Multimodal Brain Tumor Segmentation in MRI Scans,” BrainLes 2020: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, Lima, Peru, pp. 168-178, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[71] Chinmay Savadikar, Rahul Kulhalli, and Bhushan Garware, “Brain Tumour Segmentation using Probabilistic U-Net,” BrainLes 2020: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, Lima, Peru, pp. 255-264, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[72] Chen Chen et al., “3D Dilated Multi-Fiber Network for Real-Time Brain Tumor Segmentation in MRI,” MICCAI 2019: Medical Image Computing and Computer Assisted Intervention, pp. 184-192, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[73] Qiran Jia, and Hai Shu, “BiTr-Unet: A CNN-Transformer Combined Network for MRI Brain Tumor Segmentation,” BrainLes 2021: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, pp. 3-14, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[74] Md Mahfuzur Rahman Siddiquee, and Andriy Myronenko, “Redundancy Reduction in Semantic Segmentation of 3D Brain Tumor MRIs,” arXiv:2111.00742, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[75] Camillo Saueressig et al., “A Joint Graph and Image Convolution Network for Automatic Brain Tumor Segmentation,” BrainLes 2021: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, pp. 356-365, 2021.
[CrossRef] [Google Scholar] [Publisher Link]