Researchers audited the assumptions of large language model (LLM) benchmark evaluations, comparing two modalities (chat UI and API) with and without web search enabled. They found that chat UI responses were less accurate than API responses and that enabling web search reduced accuracy by up to 8 percentage points. The study highlights the need for AI safety evaluations to account for modality, multi-run consistency, search conditions, and response-level behaviors to better reflect how deployed AI systems behave in practice.