Publication Type
Working Paper
Version
publishedVersion
Publication Date
9-2023
Abstract
System logs play a critical role in maintaining the reliability of software systems. Fruitful studies have explored automatic log-based anomaly detection and achieved notable accuracy on benchmark datasets. However, when applied to large-scale cloud systems, these solutions face limitations due to high resource consumption and lack of adaptability to evolving logs. In this paper, we present an accurate, lightweight, and adaptive log-based anomaly detection framework, referred to as SeaLog. Our method introduces a Trie-based Detection Agent (TDA) that employs a lightweight, dynamically-growing trie structure for real-time anomaly detection. To enhance TDA's accuracy in response to evolving log data, we enable it to receive feedback from experts. Interestingly, our findings suggest that contemporary large language models, such as ChatGPT, can provide feedback with a level of consistency comparable to human experts, which can potentially reduce manual verification efforts. We extensively evaluate SeaLog on two public datasets and an industrial dataset. The results show that SeaLog outperforms all baseline methods in terms of effectiveness, runs 2X to 10X faster and only consumes 5% to 41% of the memory resource.
Discipline
Artificial Intelligence and Robotics | Software Engineering
Research Areas
Intelligent Systems and Optimization
Areas of Excellence
Digital transformation
First Page
1
Last Page
12
Identifier
10.48550/arXiv.2306.05032
Citation
LIU, Jinyang; HUANG, Junjie; HUO, Yintong; JIANG, Zhihan; GU, Jiazhen; CHEN, Zhuangbin; FENG, Cong; YAN, Minzhi; and LYU, R. Michael.
Log-based anomaly detection based on EVT theory with feedback. (2023). 1-12.
Available at: https://ink.library.smu.edu.sg/sis_research/11212
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
Additional URL
https://doi.org/10.48550/arXiv.2306.05032