2025年人工智能代理系统故障模式分类白皮书

[Pete Bryan, Giorgio Severi, Joris de Gruyter, Daniel Jones, Blake Bullwinkel,Amanda Minnich, Shiven Chawla, Gary Lopez, Martin Pouliot, Adam Fourney,Whitney Maxwell, Katherine Pratt, Saphir Qi, Nina Chikanov, Roman Lutz, RajaSekhar Rao Dheekonda, Bolor-Erdene Jagdagdorj, Eugenia Kim, Justin Song,Keegan Hines, Daniel Jones, Richard Lundeen, Sam Vaughan, Victoria Westerhoff,Yonatan Zunger, Chang Kawaguchi, Mark Russinovich, Ram Shankar Siva Kumar] Contents Abstract..................................................................................................................................................................... 2Introduction ............................................................................................................................................................2Agentic systems: Functionality and common patterns.....................................................................3Overview of failure modes ................................................................................................................................ 6What effects can these failure modes have? ........................................................................................7Mitigations and design considerations...................................................................................................8Limitations of our analysis ...........................................................................................................................10Case study: Memory poisoning attack on an agentic AI email assistant ....................................... 10Introduction .......................................................................................................................................................10Context and setup...........................................................................................................................................11Baseline attack description ..........................................................................................................................12Mechanism of the attack ..............................................................................................................................12Results and observations..............................................................................................................................13Challenges and mitigation strategies ......................................................................................................15Taxonomy – Details.............................................................................................................................................. 16Novel security failure modes.......................................................................................................................16Novel safety failure modes ..........................................................................................................................19Existing security failure modes...................................................................................................................21Existing safety failure modes.......................................................................................................................24Acknowledgement ............................................................................................................................................... 27Related work...........................................................................................................................................................27 Abstract Agentic AI systems are gaining prominence in both research and industry to increase the impact andvalue of generative AI. To understand the potential weaknesses in such systems and develop an approachfor testing them, Microsoft’s AI Red Team (AIRT) worked with stakeholders across the company andconducted a failure mode and effects analysis of the current and envisaged future agentic AI system In addition, there are numerous failure modes that currently affect generative AI models whoseprominence or potential impact is greatly increased when contextualized in an agentic AI system. Whilethere is still a wide degree of variance in architectural and engineering approaches for these systems, Introduction A clear understanding of the scope of agentic AI systems, both in their current form and in potentialfuture variants, is critical to effectively plan security testing and response operations. There is ongoingdebate in the industry about what exactly constitutes an agentic AI system, and for the purposes of thisanalysis, Microsoft AI Red Team (AIRT) started from the definition provided by the World Economic The Microsoft AI Red Team followed two key stages to understand current and future shape of agentic AIsystems. •First, we conducted systematic interviews with external practitioners working on developing •Next, the AI Red Team worked