Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agents
2509.26354v1
cs.AI, cs.CL, cs.LG
2025-10-02
Авторы:
Shuai Shao, Qihan Ren, Chen Qian, Boyi Wei, Dadi Guo, Jingyi Yang, Xinhao Song, Linfeng Zhang, Weinan Zhang, Dongrui Liu, Jing Shao
Abstract
Advances in Large Language Models (LLMs) have enabled a new class of
self-evolving agents that autonomously improve through interaction with the
environment, demonstrating strong capabilities. However, self-evolution also
introduces novel risks overlooked by current safety research. In this work, we
study the case where an agent's self-evolution deviates in unintended ways,
leading to undesirable or even harmful outcomes. We refer to this as
Misevolution. To provide a systematic investigation, we evaluate misevolution
along four key evolutionary pathways: model, memory, tool, and workflow. Our
empirical findings reveal that misevolution is a widespread risk, affecting
agents built even on top-tier LLMs (e.g., Gemini-2.5-Pro). Different emergent
risks are observed in the self-evolutionary process, such as the degradation of
safety alignment after memory accumulation, or the unintended introduction of
vulnerabilities in tool creation and reuse. To our knowledge, this is the first
study to systematically conceptualize misevolution and provide empirical
evidence of its occurrence, highlighting an urgent need for new safety
paradigms for self-evolving agents. Finally, we discuss potential mitigation
strategies to inspire further research on building safer and more trustworthy
self-evolving agents. Our code and data are available at
https://github.com/ShaoShuai0605/Misevolution . Warning: this paper includes
examples that may be offensive or harmful in nature.
Ссылки и действия
Дополнительные ресурсы: