Publication Type
Conference Proceeding Article
Version
acceptedVersion
Publication Date
12-2017
Abstract
Recent studies showed that single-machine graph processing systems can be as highly competitive as clusterbased approaches on large-scale problems. While several outof-core graph processing systems and computation models have been proposed, the high disk I/O overhead could significantly reduce performance in many practical cases. In this paper, we propose GraphMP to tackle big graph analytics on a single machine. GraphMP achieves low disk I/O overhead with three techniques. First, we design a vertex-centric sliding window (VSW) computation model to avoid reading and writing vertices on disk. Second, we propose a selective scheduling method to skip loading and processing unnecessary edge shards on disk. Third, we use a compressed edge cache mechanism to fully utilize the available memory of a machine to reduce the amount of disk accesses for edges. Extensive evaluations have shown that GraphMP could outperform state-of-the-art systems such as GraphChi, X-Stream and GridGraph by 31.6x, 54.5x and 23.1x respectively, when running popular graph applications on a billion-vertex graph.
Keywords
Graph Processing, Big Data, Parallel Computing
Discipline
Computer and Systems Architecture | Software Engineering
Research Areas
Software and Cyber-Physical Systems
Publication
Proceedings of the 23rd International Conference on Parallel and Distributed Systems (ICPADS): 2017 IEEE, Shenzhen, China, December 15-17
First Page
1
Last Page
10
ISBN
1521-9097
Identifier
10.1109/ICPADS.2017.00045
Publisher
IEEE
City or Country
Shenzhen, China
Citation
SUN, Peng; WEN, Yonggang; TA, Nguyen Binh Duong; and XIAO, Xiaokui.
GraphMP: an efficient semi-external-memory big graph processing system on a single machine. (2017). Proceedings of the 23rd International Conference on Parallel and Distributed Systems (ICPADS): 2017 IEEE, Shenzhen, China, December 15-17. 1-10.
Available at: https://ink.library.smu.edu.sg/sis_research/4764
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
Additional URL
https://doi.org/10.1109/ICPADS.2017.00045