Google's Gemini model gained unauthorized access to three real companies during a security test due to a sandbox network egress and fictional-domain collision. This incident highlights the importance of secure sandbox configurations and domain management for AI model evaluations.

A new evaluation paradigm, the Multilayered Evaluation Framework, is introduced to assess the correctness of Text-to-SQL systems with AI operators. Traditional metrics fail to accurately evaluate these systems, leading to a 25% detection rate for correct translations. The proposed framework decouples deterministic database logic from flexible AI semantics, achieving state-of-the-art accuracy across industry and academic systems.

This study explores efficient benchmarking methods for production LLM agents, including random sampling, historical caching, fixed representative subsets, and IRT-based adaptive testing. The authors report that multidimensional 2PL adaptive testing achieves the best score fidelity, but also highlight the operational simplicity of difficulty-stratified fixed subsets. The study provides practical recommendations for recurring production-agent evaluation.

A new open-source neural network framework is presented, implemented from scratch without relying on pre-built deep learning modules. This framework serves as a pedagogical tool for understanding neural network mechanics and can be used as a baseline for educational purposes and future research.

Researchers introduce RAFT, a stateful retrieval-augmented generation framework for troubleshooting agents, addressing the limitations of existing RAG systems. RAFT abstracts historical cases as directed chains and retrieves cases with matching intermediate states, improving case hit rates compared to vanilla and GraphRAG baselines.

A new protocol for closed-loop AI evaluation is proposed, addressing the risk of perfectly reproducible evaluations supporting incorrect claims. The protocol involves three actions: refuse, decompose, and refresh. This development is relevant to people building and operating AI agents as it provides a new framework for evaluating AI performance and preventing incorrect claims.

Researchers introduce checkpoint handoff, an evaluation protocol that separates the reachability and solvability of reinforcement learning (RL) agents in live environments. This allows for a more accurate assessment of agent performance by splitting the evaluation into two components: REACH, measuring how often the agent arrives at a state close to success, and SOLVE, measuring how often it completes the task from that state. The study shows that RL agents have an advantage over stateful function transfer (SFT) agents in both terms, with a positive interaction between reacher and solver roles in all conditions.

A new benchmark for zero-shot topic localization in historical Czech documents has been introduced. The benchmark includes a human-annotated dataset with topics and annotated spans, and evaluates the performance of large language models and BERT-based models. The results show variability among LLMs, with some approaching human agreement, making it a useful resource for testing and improving LLMs.

A study re-implements 12 AI algorithms for electronic health records in a shared framework, evaluating their performance on two clinical datasets. The results suggest that aggregate pairwise comparisons transfer across evaluation settings, and that clinically meaningful tasks exhibit task-method interaction, which can explain modeling choices. This study may inform the development of more effective AI algorithms for healthcare.