Learning visual clothing style with heterogeneous dyadic co-occurrences

Research output: Contribution to journalConference articleResearchpeer-review

With the rapid proliferation of smart mobile devices, users now take millions of photos every day. These include large numbers of clothing and accessory images. We would like to answer questions like 'What outfit goes well with this pair of shoes?' To answer these types of questions, one has to go beyond learning visual similarity and learn a visual notion of compatibility across categories. In this paper, we propose a novel learning framework to help answer these types of questions. The main idea of this framework is to learn a feature transformation from images of items into a latent space that expresses compatibility. For the feature transformation, we use a Siamese Convolutional Neural Network (CNN) architecture, where training examples are pairs of items that are either compatible or incompatible. We model compatibility based on co-occurrence in large-scale user behavior data, in particular co-purchase data from Amazon.com. To learn cross-category fit, we introduce a strategic method to sample training data, where pairs of items are heterogeneous dyads, i.e., the two elements of a pair belong to different high-level categories. While this approach is applicable to a wide variety of settings, we focus on the representative problem of learning compatible clothing style. Our results indicate that the proposed framework is capable of learning semantic information about visual style and is able to generate outfits of clothes, with items from different categories, that go well together.

Original languageEnglish
JournalProceedings of the IEEE International Conference on Computer Vision
Pages (from-to)4642-4650
Number of pages9
ISSN1550-5499
DOIs
Publication statusPublished - 17 Feb 2015
Externally publishedYes
Event15th IEEE International Conference on Computer Vision, ICCV 2015 - Santiago, Chile
Duration: 11 Dec 201518 Dec 2015

Conference

Conference15th IEEE International Conference on Computer Vision, ICCV 2015
CountryChile
CitySantiago
Period11/12/201518/12/2015

Bibliographical note

Publisher Copyright:
© 2015 IEEE.

ID: 301828880