Publication Type

Journal Article

Version

publishedVersion

Publication Date

2010

Abstract

Topic detection (TD) is a fundamental research issue in the Topic Detection and Tracking (TDT) community with practical implications; TD helps analysts to separate the wheat from the chaff among the thousands of incoming news streams. In this paper, we propose a simple and effective topic detection model called the temporal Discriminative Probabilistic Model (DPM), which is shown to be theoretically equivalent to the classic vector space model with feature selection and temporally discriminative weights. We compare DPM to its various probabilistic cousins, ranging from mixture models like von-Mises Fisher (vMF) to mixed membership models like Latent Dirichlet Allocation (LDA). Benchmark results on the TDT3 data set show that sophisticated models, such as vMF and LDA, do not necessarily lead to better results; in the case of LDA, notably worst performance was obtained under variational inference, which is likely due to the significantly large number of LDA model parameters involved for document-level topic detection. On the contrary, using a relatively simple time-aware probabilistic model such as DPM suffices for both offline and online topic detection tasks, making DPM a theoretically elegant and effective model for practical topic detection.

Keywords

DPM, TFIDF, Topic detection, bursty feature, online, probabilistic model, time-aware

Discipline

Databases and Information Systems

Publication

IEEE Transactions on Pattern Analysis Machine Intelligence

Volume

32

Issue

10

First Page

1795

Last Page

1808

ISSN

0162-8828

Identifier

10.1109/TPAMI.2009.203

Publisher

IEEE

Additional URL

http://doi.org/10.1109/TPAMI.2009.203

Download

Share

COinS