Sketch_Based Image Retrieval Using Convolutional Neural Network with Multi_Step Training
Subject Areas : Image Processing
Azita Gheitasi
1
,
Hassan Farsi
2
,
Sajad Mohamadzadeh
3
*
1 - Department of Electrical and Computer Engineering, University of Birjand, Birjand, Iran
2 - Department of Electrical and Computer Engineering, University of Birjand, Birjand, Iran
3 - Department of Electrical and Computer Engineering, University of Birjand, Birjand, Iran
Keywords: Sketch-Based Image Retrieval (SBIR), Deep Learning, Multi-step training, Contrastive loss, Triplet loss,
Abstract :
The expansion of touch-screen devices has provided the possibility of human-machine interactions in the form of free-hand drawings. In sketch-based image retrieval (SBIR) systems, the query image is a simple binary design that represents the mental image of a person with the rough shape of an object. A simple sketch is convenient and efficient for recording ideas visually, and can outdo hundreds of words. The objective is to retrieve a natural image with the same label as the query sketch. This article presents a multi-step training method. Regression functions are used in the deep network structure to improve system performance, and various loss functions are employed for a better convergence of the retrieval system. The convolutional neural network used has two branches, one related to the sketch and the other related to the image, and these two branches can have the same or different architecture. After four training steps, a 56.48% MAP was achieved, indicating the desirable performance of the network.
[1] D. Birari, D. Hiran, and V. Narawade, ''Survey on Sketch Based Image and Data Retrieval'', in ICCCE 2019 Springer, 2020, pp. 285-290.
[2] Y. Li, and W. Li, ''A survey of sketch-based image retrieval'', Machine Vision and Applications, Vol. 29, No. 7, 2018, pp. 1083-1100.
[3] S. Mohammadzadeh, and H. Farsi, ''Image retrieval using color-texture features extracted from Gabor-Walsh wavelet pyramid'', Journal of Information Systems and Telecommunication, Vol. 2, No. 1, 2014, pp. 31-40.
[4] L. Liu, F. Shen, Y. Shen, X. Liu, and L. Shao, ''Deep sketch hashing: Fast free-hand sketch-based image retrieval'', in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2862-2871.
[5] Z. Hossein-Nejad, H. Agahi, and A. Mahmoodzadeh, ''Remote Sensing Image Registration based on a Geometrical Model Matching'', Journal of Information Systems and Telecommunication (JIST), Vol. 5, No. 36, 2021, pp. 41.
[6] J. Zhang, F. Shen, L. Liu, F. Zhu, M. Yu, L. Shao, H. T. Shen, and L. Van Gool, ''Generative domain-migration hashing for sketch-to-image retrieval'', in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 297-314.
[7] A. Sain, A. K. Bhunia, Y. Yang, T. Xiang, and Y. Z. Song, ''Style me up: Towards style-agnostic sketch-based image retrieval'', in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 8504-8513.
[8] Z. Dorrani, H. Farsi, and S. Mohamadzadeh, ''Image edge detection with fuzzy ant colony optimization algorithm'', International Journal of Engineering, Vol. 33, No. 12, 2020, pp. 2464-2470.
[9] E. Askari, and S. Motamed, ''Computational Model for Image Processing in the Minds of People with Visual Agnosia using Fuzzy Cognitive Map'', Journal of Information Systems and Telecommunication (JIST), Vol. 2, No. 42, 2023, pp. 102.
[10] T. Bui, L. Ribeiro, M. Ponti and J. Collomosse, ''Sketching out the details: Sketch-based image retrieval using convolutional neural networks with multi-stage regression'', Computers & Graphics, 2018, Vol. 71, pp. 77-87. [11] H. Farsi, and S. Mohamadzadeh, ''Combining Hadamard matrix, discrete wavelet transform and DCT features based on PCA and KNN for image retrieval'', Majlesi Journal of Electrical Engineering, Vol. 7, No. 1, 2013, pp. 9-15.
[12] A. Del Bimbo, and P. Pala, ''Visual image retrieval by elastic matching of user sketches'', IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 19, No. 2, 1997, pp. 121-132.
[13] Y. Chans, Z. Lie, D. P. Lopresti, and S. Y. Kung, ''Feature-based approach for image retrieval by sketch'', in Multimedia Storage and Archiving Systems II, 1997, Vol. 3229, pp. 220-231.
[14] R. K. Rajendran, and S. F. Chang, ''Image retrieval with sketches and compositions'', in 2000 IEEE International Conference on Multimedia and Expo. ICME 2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No. 00TH8532), 2000, Vol. 2, pp. 717-720.
[15] M. Rezaei, and M. Rezaei, ''Foreground-Back ground Segmentation using K-Means Clustering Algorithm and Support Vector Machin'', Journal of Information Systems and Telecommunication (JIST), Vol. 1, No. 41, 2023, pp. 65.
[16] A. K. Bhunia, P. N. Chowdhury, A. Sain, Y. Yang, T. Xiang, and Y. Z. Song, ''More photos are all you need: Semi-supervised learning for fine-grained sketch-based image retrieval'', in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 4247-4256.
[17] S. Dey, P. Riba, A. Dutta, J. Llados, and Y. Z. Song, ''Doodle to search: Practical zero-shot sketch-based image retrieval'', in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 2179-2188.
[18] A. Gheitasi, H. Farsi, and S. Mohamadzadeh, ''Estimation of hand skeletal postures by using deep convolutional neural networks'', International Journal of Engineering, Vol. 33, No. 4, 2020, pp. 552-559.
[19] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, ''Going deeper with convolutions'', in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1-9.
[20] Q. Yu, F. Liu, Y. Z. Song, T. Xiang, T. M. Hospedales, and C. C. Loy, ''Sketch me that shoe'', in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 799-807.
[21] S. D. Bhattacharjee, J. Yuan, W. Hong, and X. Ruan, ''Query adaptive instance search using object sketches'', in Proceedings of the 24th ACM international conference on Multimedia, 2016, pp. 1306-1315.
[22] F. Wang, L. Kang, and Y. Li, ''Sketch-based 3d shape retrieval using convolutional neural networks'', in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1875-1883.
[23] Y. Qi, Y. Z. Song, H. Zhang, and J. Liu, ''Sketch-based image retrieval via siamese convolutional neural network'', in 2016 IEEE International Conference on Image Processing (ICIP), 2016, pp. 2460-2464.
[24] A. Fuentes, and J. M. Saavedra, ''Sketch-qnet: A quadruplet convnet for color sketch-based image retrieval'', in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 2134-2141.
[25] K. R. V. Vakili_Zare, and H. Rezaei, ''K_Nearest Neighbor Classification Using Data and Deep Neural Networks'', in 3rd Internattional Conference on Soft Computing, 2019, pp. 1034-1040.
[26] F. Schroff, D. Kalenichenko, and J. Philbin, ''Facenet: A unified embedding for face recognition and clustering'', in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 815-823.
[27] X. Wang, and A. Gupta, ''Unsupervised learning of visual representations using videos'', in Proceedings of the IEEE international conference on computer vision, 2015, pp. 2794-2802.
[28] A. Gordo, J. Almazan, J. Revaud, and D. Larlus, ''Deep image retrieval: Learning global representations for image search'', in European conference on computer vision, 2016, pp. 241-257.
[29] P. Sangkloy, N. Burnell, C. Ham and J. Hays, ''The sketchy database: learning to retrieve badly drawn bunnies'', ACM Transactions on Graphics (TOG), Vol. 35, No. 4, 2016, pp. 1-12.
[30] T. Bui, L. Ribeiro, M. Ponti, and J. Collomosse, ''Compact descriptors for sketch-based image retrieval using a triplet loss convolutional neural network'', Computer Vision and Image Understanding, Vol. 164, 2017, pp. 27-37.
[31] J. M. Saavedra, ''Rst-shelo: sketch-based image retrieval using sketch tokens and square root normalization'', Multimedia Tools and Applications, Vol. 76, No. 1, 2017, pp. 931-951.
[32] C. Bai, J. Chen, Q. Ma, P. Hao, and S. Chen, ''Cross-domain representation learning by domain-migration generative adversarial network for sketch-based image retrieval'', Journal of Visual Communication and Image Representation, Vol. 71, 2020, pp. 102835.
[33] a. P. R. G. G. Rajput, ''Sketch Based Image Retrieval in Large Databases Using Edge Features'', International Journal of Recent Technology and Engineering (IJRTE), Vol. 08, 2020, pp. 2277-3878.
[34] A. Dutta, and Z. Akata, ''Semantically tied paired cycle consistency for zero-shot sketch-based image retrieval'', in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 5089-5098.
[35] Q. Liu, L. Xie, H. Wang, and A. L. Yuille, ''Semantic-aware knowledge preservation for zero-shot sketch-based image retrieval'', in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 3662-3671.
[36] T. Dutta, A. Singh, and S. Biswas, ''Style guide: zero-shot sketch-based image retrieval using style-guided image generation'', IEEE Transactions on Multimedia, Vol. 23, 2020, pp. 2833-2842.
[37] P. Torres, and J. M. Saavedra, ''Compact and effective representations for sketch-based image retrieval'', in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 2115-2123. [38] R. Chavhan, ''Zero-Shot Sketch Based Image Retrieval'', INDIAN INSTITUTE OF TECHNOLOGY BOMBAY, 2021.
[39] C. Deng, X. Xu, H. Wang, M. Yang, and D. Tao, ''Progressive cross-modal semantic network for zero-shot sketch-based image retrieval'', IEEE Transactions on Image Processing, Vol. 29, 2020, pp. 8892-8902.
[40] O. Tursun, S. Denman, S. Sridharan, E. Goan, and C. Fookes, ''An efficient framework for zero-shot sketch-based image retrieval'', Pattern Recognition, Vol. 126, 2022, pp. 108528.
[41] X. Zhang, Y. Huang, Q. Zou, Y. Pei, R. Zhang, and S. Wang, ''A hybrid convolutional neural network for sketch recognition'', Pattern Recognition Letters, Vol. 130, 2020, pp. 73-82.
[42] A. K. Bhunia, Y. Yang, T. M. Hospedales, T. Xiang, and, Y. Z. Song, ''Sketch less for more: On-the-fly fine-grained sketch-based image retrieval'', in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 9779-9788.