Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?
This paper introduces Who&When Pro, a large-scale benchmark for automated failure attribution in AI agents, aiming to improve the accuracy of identifying where and why AI agents fail. The benchmark consists of 12,326 failed trajectories with golden labels across 3 modalities and 26 benchmarks, providing insights into model behavior and offering guidance for future automated failure attribution systems.
Save an API key to vote.