ML Anthology

EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities

Talor Abramovich, Meet Udeshi, Minghao Shao, Kilian Lieret, Haoran Xi, Kimberly Milner, Sofija Jancheska, John Yang, Carlos E Jimenez, Farshad Khorrami, Prashanth Krishnamurthy, Brendan Dolan-Gavitt, Muhammad Shafique, Karthik R Narasimhan, Ramesh Karri, Ofir Press
ICML 2025 pp. 246-355
/icml/2025/abramovich2025icml-enigma/

Abstract

Although language model (LM) agents have demonstrated increased performance in multiple domains, including coding and web-browsing, their success in cybersecurity has been limited. We present EnIGMA, an LM agent for autonomously solving Capture The Flag (CTF) challenges. We introduce new tools and interfaces to improve the agent’s ability to find and exploit security vulnerabilities, focusing on interactive terminal programs. These novel Interactive Agent Tools enable LM agents, for the first time, to run interactive utilities, such as a debugger and a server connection tool, which are essential for solving these challenges. Empirical analysis on 390 CTF challenges across four benchmarks demonstrate that these new tools and interfaces substantially improve our agent’s performance, achieving state-of-the-art results on NYU CTF, Intercode-CTF, and CyBench. Finally, we analyze data leakage, developing new methods to quantify it and identifying a new phenomenon we term soliloquizing, where the model self-generates hallucinated observations without interacting with the environment.

PDF ICML OpenReview Semantic Scholar

Cite

Text

Abramovich et al. "EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities." Proceedings of the 42nd International Conference on Machine Learning, 2025.

Markdown

[Abramovich et al. "EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities." Proceedings of the 42nd International Conference on Machine Learning, 2025.](https://mlanthology.org/icml/2025/abramovich2025icml-enigma/)

BibTeX

@inproceedings{abramovich2025icml-enigma,
  title     = {{EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities}},
  author    = {Abramovich, Talor and Udeshi, Meet and Shao, Minghao and Lieret, Kilian and Xi, Haoran and Milner, Kimberly and Jancheska, Sofija and Yang, John and Jimenez, Carlos E and Khorrami, Farshad and Krishnamurthy, Prashanth and Dolan-Gavitt, Brendan and Shafique, Muhammad and Narasimhan, Karthik R and Karri, Ramesh and Press, Ofir},
  booktitle = {Proceedings of the 42nd International Conference on Machine Learning},
  year      = {2025},
  pages     = {246-355},
  volume    = {267},
  url       = {https://mlanthology.org/icml/2025/abramovich2025icml-enigma/}
}