Feature selection (FS) is an important technique in machine learning and data mining, especially for largescale high-dimensional data. Most existing studies have been restricted to batch learning, which is often inefficient and poorly scalable when handling big data in real world. As real data may arrive sequentially and continuously, batch learning has to retrain the model for the new coming data, which is very computationally intensive. Online feature selection (OFS) is a promising new paradigm that is more efficient and scalable than batch learning algorithms. However, existing online algorithms usually fall short in their inferior efficacy. In this article, we present a novel second-order OFS algorithm that is simple yet effective, very fast and extremely scalable to deal with large-scale ultra-high dimensional sparse data streams. The basic idea is to exploit the second-order information to choose the subset of important features with high confidence weights. Unlike existing OFS methods that often suffer from extra high computational cost, we devise a novel algorithm with a MaxHeap-based approach, which is not only more effective than the existing firstorder algorithms, but also significantly more efficient and scalable. Our extensive experiments validated that the proposed technique achieves highly competitive accuracy as compared with state-of-The-Art batch FS methods, meanwhile it consumes significantly less computational cost that is orders of magnitude lower. Impressively, on a billion-scale synthetic dataset (1-billion dimensions, 1-billion non-zero features, and 1- million samples), the proposed algorithm takes less than 3 minutes to run on a single PC.
Feature selection, Second-order online learning, Sparsity, Ultra-high dimensionality
Databases and Information Systems | Numerical Analysis and Computation
Data Management and Analytics
ACM Transactions on Knowledge Discovery from Data
Association for Computing Machinery (ACM)
WU, Yue; HOI, Steven C. H.; MEI, Tao; and Yu, Nenghai.
Large-scale online feature selection for ultra-high dimensional sparse data. (2017). ACM Transactions on Knowledge Discovery from Data. 11, (4),. Research Collection School Of Information Systems.
Available at: http://ink.library.smu.edu.sg/sis_research/3781
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 License.