Sponsor
This work was partially done when the first author was an intern at Adobe Research. This work was supported in part by NSF IIS-1321119.
Document Type
Conference Proceeding
Publication Date
7-2017
Subjects
Neural networks -- Applications to image retrieval, Image processing -- Digital techniques, Pattern recognition systems
Abstract
The performance of image retrieval has been improved tremendously in recent years through the use of deep feature representations. Most existing methods, however, aim to retrieve images that are visually similar or semantically relevant to the query, irrespective of spatial configuration. In this paper, we develop a spatial-semantic image search technology that enables users to search for images with both semantic and spatial constraints by manipulating concept text-boxes on a 2D query canvas. We train a convolutional neural network to synthesize appropriate visual features that captures the spatial-semantic constraints from the user canvas query. We directly optimize the retrieval performance of the visual features when training our deep neural network. These visual features then are used to retrieve images that are both spatially and semantically relevant to the user query. The experiments on large-scale datasets such as MS-COCO and Visual Genome show that our method outperforms other baseline and state-of-the-art methods in spatial-semantic image search.
Persistent Identifier
http://archives.pdx.edu/ds/psu/24048
Citation Details
Long Mai, Hailin Jin, Zhe Lin, Chen Fang, Jonathan Brandt, Feng Liu; The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 4718-4727
Description
This CVPR 2017 paper is the Open Access version provided by the Computer Vision Foundation.
Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright.