LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs

Weibin Meng, Ying Liu, Yichen Zhu, Shenglin Zhang, Dan Pei, Yuqing Liu, Yihao Chen, Ruizhi Zhang, Shimin Tao, Pei Sun, Rong Zhou

IJCAI 2019 pp. 4739-4745

doi:10.24963/IJCAI.2019/658 /ijcai/2019/meng2019ijcai-loganomaly/

Abstract

Recording runtime status via logs is common for almost every computer system, and detecting anomalies in logs is crucial for timely identifying malfunctions of systems. However, manually detecting anomalies for logs is time-consuming, error-prone, and infeasible. Existing automatic log anomaly detection approaches, using indexes rather than semantics of log templates, tend to cause false alarms. In this work, we propose LogAnomaly, a framework to model unstructured a log stream as a natural language sequence. Empowered by template2vec, a novel, simple yet effective method to extract the semantic information hidden in log templates, LogAnomaly can detect both sequential and quantitive log anomalies simultaneously, which were not done by any previous work. Moreover, LogAnomaly can avoid the false alarms caused by the newly appearing log templates between periodic model retrainings. Our evaluation on two public production log datasets show that LogAnomaly outperforms existing log-based anomaly detection methods.

PDF IJCAI Semantic Scholar

Cite

Text

Meng et al. "LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs." International Joint Conference on Artificial Intelligence, 2019. doi:10.24963/IJCAI.2019/658

Markdown

[Meng et al. "LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs." International Joint Conference on Artificial Intelligence, 2019.](https://mlanthology.org/ijcai/2019/meng2019ijcai-loganomaly/) doi:10.24963/IJCAI.2019/658

BibTeX

@inproceedings{meng2019ijcai-loganomaly,
  title     = {{LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs}},
  author    = {Meng, Weibin and Liu, Ying and Zhu, Yichen and Zhang, Shenglin and Pei, Dan and Liu, Yuqing and Chen, Yihao and Zhang, Ruizhi and Tao, Shimin and Sun, Pei and Zhou, Rong},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year      = {2019},
  pages     = {4739-4745},
  doi       = {10.24963/IJCAI.2019/658},
  url       = {https://mlanthology.org/ijcai/2019/meng2019ijcai-loganomaly/}
}