On automatic identification of retail products in images of racks in the supermarkets/ Bikash Santra

Thesis (Ph.D.) - Indian Statistical Institute, 2021

Includes bibliographical references

1 Introduction -- 2 An End-to-End Annotation-free Machine Vision System -- 3 Fine-grained Classification of Products -- 4 Graph-based Non-maximal Suppression of Region Proposals -- 5 Identification of Empty Spaces on Shelves -- 6 Conclusions -- A Evaluation Indicators for Measuring Product Detection Performance -- B Datasets of Retail Products -- C Assumptions and Proof of the Theorem 3.1 --

Guided by Prof. Dipti Prasad Mukherjee

An image of a rack in a supermarket displays a number of retail products. The identification and localization of these individual products from the images of racks is a challenge for any machine vision system. In this thesis, we aim to address this problem and suggest a set of computer vision based solutions for automatic identification of these retail products. We design a novel classifier that differentiates the similarly looking yet non-identical (fine-grained) products for improving the performance of our machine
vision system. The proposed fine-grained classifier simultaneously captures both object-level and partlevel (image of an object consists of multiple parts or image patches) cues of the products for accurately distinguishing the fine-grained products. A graph-based non-maximal suppression strategy is proposed that selects a winner region proposal among a group of proposals representing a product. This solves an important bottleneck of conventional greedy non-maximal suppression algorithm for disambiguation of overlapping region proposals generated in an intermediate step of our proposed system. We initiate the solution of the problem of automatic product identification by developing an end-to-end annotation-free machine vision system for recognition and localization of products on the rack. The proposed system introduces a novel exemplar-driven region proposal strategy that overcomes the shortcomings of traditional exemplar-independent region proposal schemes like selective window search. Finally, we find the empty spaces (or gaps between products) in each shelf of any rack by creating a graph of superpixels for the rack. We extract the visual features of superpixels from our graph convolutional and Siamese networks. Subsequently, we send the graph along with the features of superpixels to a structural support vector machine for discovering the empty spaces of the shelves. The efficacy of the proposed approaches are established through various experiments on our In-house dataset and three publicly available benchmark datasets: Grozi-120 [Merler et al., IEEE CVPR 2007, 1-8], Grocery Products [George et al., Springer ECCV 2014, 440-455], and WebMarket [Zhang et al., Springer ACCV 2007, 800-810].