A Spectroscopy of Texts for Effective Clustering
Conference Proceeding Article
For many clustering algorithms, such as k-means, EM, and CLOPE, there is usually a requirement to set some parameters. Often, these parameters directly or indirectly control the number of clusters to return. In the presence of different data characteristics and analysis contexts, it is often difficult for the user to estimate the number of clusters in the data set. This is especially true in text collections such as Web documents, images or biological data. The fundamental question this paper addresses is: ldquoHow can we effectively estimate the natural number of clusters in a given text collection?rdquo. We propose to use spectral analysis, which analyzes the eigenvalues (not eigenvectors) of the collection, as the solution to the above. We first present the relationship between a text collection and its underlying spectra. We then show how the answer to this question enhances the clustering process. Finally, we conclude with empirical results and related work.
Databases and Information Systems | Numerical Analysis and Scientific Computing
Data Management and Analytics
Knowledge Discovery in Databases: PKDD 2004: 8th European Conference on Principles and Practice of Knowledge Discovery in Databases, Pisa, Italy, September 20-24: Proceedings
City or Country
LI, Wenyuan; NG, Wee-Keong; ONG, Kok-Leong; and LIM, Ee Peng.
A Spectroscopy of Texts for Effective Clustering. (2004). Knowledge Discovery in Databases: PKDD 2004: 8th European Conference on Principles and Practice of Knowledge Discovery in Databases, Pisa, Italy, September 20-24: Proceedings. 3202, 301-312. Research Collection School Of Information Systems.
Available at: http://ink.library.smu.edu.sg/sis_research/1018