A new benchmark, CABRA, is proposed to evaluate the abilities of coding agents, particularly their ability to understand code. CABRA generates tasks from scratch and scales difficulty in various ways, demonstrating that coding agents can perform well by offloading work to tools, but struggle with tasks that require intense understanding of code.