Pose2Instance: Harnessing Keypoints for Person Instance Segmentation
Research output: Working paper › Preprint › Research
Standard
Pose2Instance: Harnessing Keypoints for Person Instance Segmentation. / Belongie, Serge; Tripathi, Subarna; Collins, Maxwell D.; Brown, Matthew.
2022.Research output: Working paper › Preprint › Research
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - UNPB
T1 - Pose2Instance: Harnessing Keypoints for Person Instance Segmentation
AU - Belongie, Serge
AU - Tripathi, Subarna
AU - Collins, Maxwell D.
AU - Brown, Matthew
PY - 2022
Y1 - 2022
N2 - Human keypoints are a well-studied representation of people.We explore how to use keypoint models to improve instance-level person segmentation. The main idea is to harness the notion of a distance transform of oracle provided keypoints or estimated keypoint heatmaps as a prior for person instance segmentation task within a deep neural network. For training and evaluation, we consider all those images from COCO where both instance segmentation and human keypoints annotations are available. We first show how oracle keypoints can boost the performance of existing human segmentation model during inference without any training. Next, we propose a framework to directly learn a deep instance segmentation model conditioned on human pose. Experimental results show that at various Intersection Over Union (IOU) thresholds, in a constrained environment with oracle keypoints, the instance segmentation accuracy achieves 10% to 12% relative improvements over a strong baseline of oracle bounding boxes. In a more realistic environment, without the oracle keypoints, the proposed deep person instance segmentation model conditioned on human pose achieves 3.8% to 10.5% relative improvements comparing with its strongest baseline of a deep network trained only for segmentation.
AB - Human keypoints are a well-studied representation of people.We explore how to use keypoint models to improve instance-level person segmentation. The main idea is to harness the notion of a distance transform of oracle provided keypoints or estimated keypoint heatmaps as a prior for person instance segmentation task within a deep neural network. For training and evaluation, we consider all those images from COCO where both instance segmentation and human keypoints annotations are available. We first show how oracle keypoints can boost the performance of existing human segmentation model during inference without any training. Next, we propose a framework to directly learn a deep instance segmentation model conditioned on human pose. Experimental results show that at various Intersection Over Union (IOU) thresholds, in a constrained environment with oracle keypoints, the instance segmentation accuracy achieves 10% to 12% relative improvements over a strong baseline of oracle bounding boxes. In a more realistic environment, without the oracle keypoints, the proposed deep person instance segmentation model conditioned on human pose achieves 3.8% to 10.5% relative improvements comparing with its strongest baseline of a deep network trained only for segmentation.
UR - https://vision.cornell.edu/se3/wp-content/uploads/2017/05/pose2instance-arxiv.pdf
U2 - https://doi.org/10.48550/arXiv.1704.01152
DO - https://doi.org/10.48550/arXiv.1704.01152
M3 - Preprint
BT - Pose2Instance: Harnessing Keypoints for Person Instance Segmentation
ER -
ID: 307527307