Publication Type
Journal Article
Version
acceptedVersion
Publication Date
5-2014
Abstract
Detecting anomaly collections is an important task with many applications, including spam and fraud detection. In an anomaly collection, entities often operate in collusion and hold different agendas to normal entities. As a result, they usually manifest collective extreme traits, i.e., members of an anomaly collection are consistently clustered toward the top or bottom ranks on certain features. We therefore propose to detect these anomaly collections by extreme feature ranks. We introduce a novel anomaly definition called Extreme Rank Anomalous Collection or ERAC. We propose a new measure of anomalousness capturing collective extreme traits based on a statistical model. As there can be a large number of ERACs of various sizes, for simplicity, we first investigate the ERAC detection problem of finding top-KERACs of a predefined size limit. We then tackle the follow-up ERAC expansion problem of uncovering the supersets of the detected ERACs that are more anomalous without any size constraint. Algorithms are proposed for both ERAC detection and expansion problems, followed by studies of their performance in four datasets. Specifically, in synthetic datasets, both ERAC detection and expansion algorithms demonstrate high precisions and recalls. In a web spam dataset, both ERAC detection and expansion algorithms discover web spammers with higher precisions than existing approaches. In an IMDB dataset, both ERAC detection and expansion algorithms identify unusual actor collections that are not easily identified by clustering-based methods. In a Chinese online forum dataset, our ERAC detection algorithm identifies suspicious “water army” spammer collections agreed by human evaluators. ERAC expansion algorithm successfully reveals two larger spammer collections with different spamming behaviors.
Keywords
Anomaly collection, Extreme feature rank, Anomaly cluster, Outlier group, Spam detection, Spam cluster
Discipline
Computer Sciences | Databases and Information Systems
Publication
Data Mining and Knowledge Discovery
Volume
29
Issue
3
First Page
689
Last Page
731
ISSN
1384-5810
Identifier
10.1007/s10618-014-0360-3
Publisher
Springer Verlag
Citation
DAI, Hanbo; ZHU, Feida; LIM, Ee Peng; and PANG, Hwee Hwa.
Detecting Anomaly Collections using Extreme Feature Ranks. (2014). Data Mining and Knowledge Discovery. 29, (3), 689-731.
Available at: https://ink.library.smu.edu.sg/sis_research/2534
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
Additional URL
https://doi.org/10.1007/s10618-014-0360-3