On Optimizing Human-Machine Task Assignments

Research output: Working paper › Preprint › Research

Standard

On Optimizing Human-Machine Task Assignments. / Belongie, Serge; Veit, Andreas; Wilber, Michael J.; Vaish, Rajan; Davis, James; Anand, Vishal; Aviral, Anshu; Chakrabarty, Prithvijit; Chandak, Yash; Chaturvedi, Sidharth; Devaraj, Chinmaya; Dhall, Ankit; Dwivedi, Utkarsh; Gupte, Sanket; Sridhar, Sharath N.; Paga, Karthik; Pahuja, Anuj; Raisinghani, Aditya; Sharma, Ayush; Sharma, Shweta; Sinha, Darpana; Thakkar, Nisarg; Vignesh, K. Bala; Verma, Utkarsh; Abhishek, Kanniganti; Agrawal, Amod; Aishwarya, Arya; Bhattacharjee, Aurgho; Dhanasekar, Sarveshwaran; Gullapalli, Venkata Karthik; Gupta, Shuchita; Chandana, G; Jain, Kinjal; Kapur, Simran; Kasula, Meghana; Kumar, Shashi; Kundaliya, Parth; Mathur, Utkarsh; Mishra, Alankrit; Mudgal, Aayush; Nadimpalli, Aditya; Nihit, Munakala Sree; Periwal, Akanksha; Sagar, Ayush; Shah, Ayush; Sharma, Vikas; Sharma, Yashovardhan; Siddiqui, Faizal; Singh, Virender; Abhinav, S.; Tambwekar, Pradyumna; Taskin, Rashida; Tripathi, Ankit; Yadav, Anurag D.

2015.

Research output: Working paper › Preprint › Research

Harvard

Belongie, S, Veit, A, Wilber, MJ, Vaish, R, Davis, J, Anand, V, Aviral, A, Chakrabarty, P, Chandak, Y, Chaturvedi, S, Devaraj, C, Dhall, A, Dwivedi, U, Gupte, S, Sridhar, SN, Paga, K, Pahuja, A, Raisinghani, A, Sharma, A, Sharma, S, Sinha, D, Thakkar, N, Vignesh, KB, Verma, U, Abhishek, K, Agrawal, A, Aishwarya, A, Bhattacharjee, A, Dhanasekar, S, Gullapalli, VK, Gupta, S, Chandana, G, Jain, K, Kapur, S, Kasula, M, Kumar, S, Kundaliya, P, Mathur, U, Mishra, A, Mudgal, A, Nadimpalli, A, Nihit, MS, Periwal, A, Sagar, A, Shah, A, Sharma, V, Sharma, Y, Siddiqui, F, Singh, V, Abhinav, S, Tambwekar, P, Taskin, R, Tripathi, A & Yadav, AD 2015 'On Optimizing Human-Machine Task Assignments'. https://doi.org/10.48550/arXiv.1509.07543

APA

Belongie, S., Veit, A., Wilber, M. J., Vaish, R., Davis, J., Anand, V., Aviral, A., Chakrabarty, P., Chandak, Y., Chaturvedi, S., Devaraj, C., Dhall, A., Dwivedi, U., Gupte, S., Sridhar, S. N., Paga, K., Pahuja, A., Raisinghani, A., Sharma, A., ... Yadav, A. D. (2015). On Optimizing Human-Machine Task Assignments. https://doi.org/10.48550/arXiv.1509.07543

Vancouver

Belongie S, Veit A, Wilber MJ, Vaish R, Davis J, Anand V et al. On Optimizing Human-Machine Task Assignments. 2015 Sep 24. https://doi.org/10.48550/arXiv.1509.07543

Author

Belongie, Serge ; Veit, Andreas ; Wilber, Michael J. ; Vaish, Rajan ; Davis, James ; Anand, Vishal ; Aviral, Anshu ; Chakrabarty, Prithvijit ; Chandak, Yash ; Chaturvedi, Sidharth ; Devaraj, Chinmaya ; Dhall, Ankit ; Dwivedi, Utkarsh ; Gupte, Sanket ; Sridhar, Sharath N. ; Paga, Karthik ; Pahuja, Anuj ; Raisinghani, Aditya ; Sharma, Ayush ; Sharma, Shweta ; Sinha, Darpana ; Thakkar, Nisarg ; Vignesh, K. Bala ; Verma, Utkarsh ; Abhishek, Kanniganti ; Agrawal, Amod ; Aishwarya, Arya ; Bhattacharjee, Aurgho ; Dhanasekar, Sarveshwaran ; Gullapalli, Venkata Karthik ; Gupta, Shuchita ; Chandana, G ; Jain, Kinjal ; Kapur, Simran ; Kasula, Meghana ; Kumar, Shashi ; Kundaliya, Parth ; Mathur, Utkarsh ; Mishra, Alankrit ; Mudgal, Aayush ; Nadimpalli, Aditya ; Nihit, Munakala Sree ; Periwal, Akanksha ; Sagar, Ayush ; Shah, Ayush ; Sharma, Vikas ; Sharma, Yashovardhan ; Siddiqui, Faizal ; Singh, Virender ; Abhinav, S. ; Tambwekar, Pradyumna ; Taskin, Rashida ; Tripathi, Ankit ; Yadav, Anurag D. / On Optimizing Human-Machine Task Assignments. 2015.

Bibtex

@techreport{0e635bcf445a4b7c9356f2b6ca5d26b3,
title = "On Optimizing Human-Machine Task Assignments",
abstract = "When crowdsourcing systems are used in combination with machine inference systems in the real world, they benefit the most when the machine system is deeply integrated with the crowd workers. However, if researchers wish to integrate the crowd with {"}off-the-shelf{"} machine classifiers, this deep integration is not always possible. This work explores two strategies to increase accuracy and decrease cost under this setting. First, we show that reordering tasks presented to the human can create a significant accuracy improvement. Further, we show that greedily choosing parameters to maximize machine accuracy is sub-optimal, and joint optimization of the combined system improves performance.",
author = "Serge Belongie and Andreas Veit and Wilber, {Michael J.} and Rajan Vaish and James Davis and Vishal Anand and Anshu Aviral and Prithvijit Chakrabarty and Yash Chandak and Sidharth Chaturvedi and Chinmaya Devaraj and Ankit Dhall and Utkarsh Dwivedi and Sanket Gupte and Sridhar, {Sharath N.} and Karthik Paga and Anuj Pahuja and Aditya Raisinghani and Ayush Sharma and Shweta Sharma and Darpana Sinha and Nisarg Thakkar and Vignesh, {K. Bala} and Utkarsh Verma and Kanniganti Abhishek and Amod Agrawal and Arya Aishwarya and Aurgho Bhattacharjee and Sarveshwaran Dhanasekar and Gullapalli, {Venkata Karthik} and Shuchita Gupta and G Chandana and Kinjal Jain and Simran Kapur and Meghana Kasula and Shashi Kumar and Parth Kundaliya and Utkarsh Mathur and Alankrit Mishra and Aayush Mudgal and Aditya Nadimpalli and Nihit, {Munakala Sree} and Akanksha Periwal and Ayush Sagar and Ayush Shah and Vikas Sharma and Yashovardhan Sharma and Faizal Siddiqui and Virender Singh and S. Abhinav and Pradyumna Tambwekar and Rashida Taskin and Ankit Tripathi and Yadav, {Anurag D.}",
year = "2015",
month = sep,
day = "24",
doi = "https://doi.org/10.48550/arXiv.1509.07543",
language = "English",
type = "WorkingPaper",

}

RIS

TY - UNPB

T1 - On Optimizing Human-Machine Task Assignments

AU - Belongie, Serge

AU - Veit, Andreas

AU - Wilber, Michael J.

AU - Vaish, Rajan

AU - Davis, James

AU - Anand, Vishal

AU - Aviral, Anshu

AU - Chakrabarty, Prithvijit

AU - Chandak, Yash

AU - Chaturvedi, Sidharth

AU - Devaraj, Chinmaya

AU - Dhall, Ankit

AU - Dwivedi, Utkarsh

AU - Gupte, Sanket

AU - Sridhar, Sharath N.

AU - Paga, Karthik

AU - Pahuja, Anuj

AU - Raisinghani, Aditya

AU - Sharma, Ayush

AU - Sharma, Shweta

AU - Sinha, Darpana

AU - Thakkar, Nisarg

AU - Vignesh, K. Bala

AU - Verma, Utkarsh

AU - Abhishek, Kanniganti

AU - Agrawal, Amod

AU - Aishwarya, Arya

AU - Bhattacharjee, Aurgho

AU - Dhanasekar, Sarveshwaran

AU - Gullapalli, Venkata Karthik

AU - Gupta, Shuchita

AU - Chandana, G

AU - Jain, Kinjal

AU - Kapur, Simran

AU - Kasula, Meghana

AU - Kumar, Shashi

AU - Kundaliya, Parth

AU - Mathur, Utkarsh

AU - Mishra, Alankrit

AU - Mudgal, Aayush

AU - Nadimpalli, Aditya

AU - Nihit, Munakala Sree

AU - Periwal, Akanksha

AU - Sagar, Ayush

AU - Shah, Ayush

AU - Sharma, Vikas

AU - Sharma, Yashovardhan

AU - Siddiqui, Faizal

AU - Singh, Virender

AU - Abhinav, S.

AU - Tambwekar, Pradyumna

AU - Taskin, Rashida

AU - Tripathi, Ankit

AU - Yadav, Anurag D.

PY - 2015/9/24

Y1 - 2015/9/24

N2 - When crowdsourcing systems are used in combination with machine inference systems in the real world, they benefit the most when the machine system is deeply integrated with the crowd workers. However, if researchers wish to integrate the crowd with "off-the-shelf" machine classifiers, this deep integration is not always possible. This work explores two strategies to increase accuracy and decrease cost under this setting. First, we show that reordering tasks presented to the human can create a significant accuracy improvement. Further, we show that greedily choosing parameters to maximize machine accuracy is sub-optimal, and joint optimization of the combined system improves performance.

AB - When crowdsourcing systems are used in combination with machine inference systems in the real world, they benefit the most when the machine system is deeply integrated with the crowd workers. However, if researchers wish to integrate the crowd with "off-the-shelf" machine classifiers, this deep integration is not always possible. This work explores two strategies to increase accuracy and decrease cost under this setting. First, we show that reordering tasks presented to the human can create a significant accuracy improvement. Further, we show that greedily choosing parameters to maximize machine accuracy is sub-optimal, and joint optimization of the combined system improves performance.

UR - https://vision.cornell.edu/se3/wp-content/uploads/2015/10/1509.07543v1.pdf

U2 - https://doi.org/10.48550/arXiv.1509.07543

DO - https://doi.org/10.48550/arXiv.1509.07543

M3 - Preprint

BT - On Optimizing Human-Machine Task Assignments

ER -

ID: 307530275