Researchers studied the reliability of agent evaluation in a Large Language Model (LLM) benchmark, identifying three key issues: 1. Oracle failures: A systematic approach detected oracle defects in 54 out of 57 variant-target combinations, leading to incorrect agent scores. In one case, the oracle reset the database to create a ground-truth hash, but failed to restore it, causing a 29.8% drop in agent success rate from 94.3% to 65.5%. 2. Configuration-dependent measurement: The study showed that the measurement apparatus is sensitive to local changes, with a 3,240-episode ablation study revealing a language effect and a second task domain reversing the effect. 3. Lack of transferability: The approach did not generalize to an independent benchmark, highlighting the need for a clear contract between the evaluation harness and the agent being evaluated.