TY - JOUR

T1 - The ignorant led by the blind

T2 - A hybrid human-machine vision system for fine-grained categorization

AU - Branson, Steve

AU - Van Horn, Grant

AU - Wah, Catherine

AU - Perona, Pietro

AU - Belongie, Serge

PY - 2014/5

Y1 - 2014/5

N2 - We present a visual recognition system for fine-grained visual categorization. The system is composed of a human and a machine working together and combines the complementary strengths of computer vision algorithms and (non-expert) human users. The human users provide two heterogeneous forms of information object part clicks and answers to multiple choice questions. The machine intelligently selects the most informative question to pose to the user in order to identify the object class as quickly as possible. By leveraging computer vision and analyzing the user responses, the overall amount of human effort required, measured in seconds, is minimized. Our formalism shows how to incorporate many different types of computer vision algorithms into a human-in-the-loop framework, including standard multiclass methods, part-based methods, and localized multiclass and attribute methods. We explore our ideas by building a field guide for bird identification. The experimental results demonstrate the strength of combining ignorant humans with poor-sighted machines the hybrid system achieves quick and accurate bird identification on a dataset containing 200 bird species.

AB - We present a visual recognition system for fine-grained visual categorization. The system is composed of a human and a machine working together and combines the complementary strengths of computer vision algorithms and (non-expert) human users. The human users provide two heterogeneous forms of information object part clicks and answers to multiple choice questions. The machine intelligently selects the most informative question to pose to the user in order to identify the object class as quickly as possible. By leveraging computer vision and analyzing the user responses, the overall amount of human effort required, measured in seconds, is minimized. Our formalism shows how to incorporate many different types of computer vision algorithms into a human-in-the-loop framework, including standard multiclass methods, part-based methods, and localized multiclass and attribute methods. We explore our ideas by building a field guide for bird identification. The experimental results demonstrate the strength of combining ignorant humans with poor-sighted machines the hybrid system achieves quick and accurate bird identification on a dataset containing 200 bird species.

KW - Attributes

KW - Birds

KW - Crowdsourcing

KW - Deformable part models

KW - Fine-grained categorization

KW - Human-in-the-loop

KW - Information gain

KW - Interactive

KW - Object recognition

KW - Parts

KW - Pose mixture models

UR - http://www.scopus.com/inward/record.url?scp=84900864212&partnerID=8YFLogxK

U2 - 10.1007/s11263-014-0698-4

DO - 10.1007/s11263-014-0698-4

M3 - Journal article

AN - SCOPUS:84900864212

VL - 108

SP - 3

EP - 29

JO - International Journal of Computer Vision

JF - International Journal of Computer Vision

SN - 0920-5691

IS - 1-2

ER -