TopicSketch: Real-time Bursty Topic Detection from Twitter
Publication Type
Journal Article
Version
acceptedVersion
Publication Date
8-2016
Abstract
Twitter has become one of the largest microblogging platforms for users around the world to share anything happening around them with friends and beyond. A bursty topic in Twitter is one that triggers a surge of relevant tweets within a short period of time, which often reflects important events of mass interest. How to leverage Twitter for early detection of bursty topics has therefore become an important research problem with immense practical value. Despite the wealth of research work on topic modelling and analysis in Twitter, it remains a challenge to detect bursty topics in real-time. As existing methods can hardly scale to handle the task with the tweet stream in real-time, we propose in this paper TopicSketch, a sketch-based topic model together with a set of techniques to achieve real-time detection. We evaluate our solution on a tweet stream with over 30 million tweets. Our experiment results show both efficiency and effectiveness of our approach. Especially it is also demonstrated that TopicSketch on a single machine can potentially handle hundreds of millions tweets per day, which is on the same scale of the total number of daily tweets in Twitter, and present bursty events in finer-granularity.
Keywords
Realtime, TopicSketch, Tweet stream, bursty topic, Twitter, Real-time systems
Discipline
Databases and Information Systems | Numerical Analysis and Scientific Computing | Social Media
Research Areas
Data Science and Engineering
Publication
IEEE Transactions on Knowledge and Data Engineering
Volume
28
Issue
8
First Page
2216
Last Page
2229
ISSN
1041-4347
Identifier
10.1109/TKDE.2016.2556661
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Citation
XIE, Wei; ZHU, Feida; Jing JIANG; LIM, Ee-Peng; and WANG, Ke.
TopicSketch: Real-time Bursty Topic Detection from Twitter. (2016). IEEE Transactions on Knowledge and Data Engineering. 28, (8), 2216-2229.
Available at: https://ink.library.smu.edu.sg/sis_research/3200
Copyright Owner and License
Authors
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
Additional URL
https://doi.org/10.1109/TKDE.2016.2556661