Title

Active Crowdsourcing for Annotation

Publication Type

Conference Proceeding Article

Publication Date

12-2015

Abstract

Crowdsourcing has shown great potential in obtaining large-scale and cheap labels for different tasks. However, obtaining reliable labels is challenging due to several reasons, such as noisy annotators, limited budget and so on. The state-of-the-art approaches, either suffer in some noisy scenarios, or rely on unlimited resources to acquire reliable labels. In this article, we adopt the learning with expert~(AKA worker in crowdsourcing) advice framework to robustly infer accurate labels by considering the reliability of each worker. However, in order to accurately predict the reliability of each worker, traditional learning with expert advice will consult with external oracles~(AKA domain experts) on the true label of each instance. To reduce the cost of consultation, we proposed two active learning approaches, margin-based and weighted difference of advices based. Meanwhile, to address the problem of limited annotation budget, we proposed a reliability-based assigning approach which actively decides who to annotate the next instance based on each worker's cumulative performance. The experimental results both on real and simulated datasets show that our algorithms can achieve robust and promising performance both in the normal and noisy scenarios with limited budget.

Keywords

Active Learning, Crowdsourcing, Online Learning

Discipline

Databases and Information Systems

Research Areas

Data Management and Analytics

Publication

IEEE/WIC/ACM Web Intelligence 2015 and IEEE/WIC/ACM Intelligent Agent Technology 2015 (WI-IAT 2015): Proceedings: December 6-8, Singapore

First Page

1

Last Page

8

Identifier

10.1109/WI-IAT.2015.34

Publisher

IEEE

City or Country

Piscataway, NJ

Additional URL

http://dx.doi.org/10.1109/WI-IAT.2015.34

This document is currently not available here.

Share

COinS