The MAMA-MIA Challenge aims to advance generalizability and fairness in breast MRI tumor segmentation and treatment response prediction by providing a standardized benchmark. It evaluates AI models using a unified scoring framework and public resources, promoting the development of robust and equitable AI systems for breast cancer imaging.

A new dataset and benchmark for evaluating large language models' understanding of competitive Chinese-language debate has been introduced. The dataset contains 148 matches, 2,698 stages, and 20,542 exchange units, with manually verified transcripts and segmentation. The benchmark provides a testbed for studying large language models' understanding of interactive argumentation and their agreement with professional judges.

VidOmni-Bench is a new benchmark for evaluating fine-grained video understanding in Video Large Language Models (Video-LLMs). It assesses whether models can accurately verify events in video captions, revealing weaknesses in current Video-LLMs that generate hallucinated descriptions and struggle with detecting incorrect event descriptions.

A new video question answering benchmark, AgentVidBench, is introduced to evaluate the spatial, temporal, and causal reasoning capabilities of MLLM agents. The benchmark provides step-by-step solution traces to assess whether agents acquire evidence to justify their answers. Experiments with 12 MLLMs show that integrating these models into agentic workflows improves performance on AgentVidBench.

Researchers introduced PolyBridgeBench, a benchmark for multimodal large language models (LLMs) to design and repair physics-grounded bridge structures. The benchmark tests LLMs' ability to generate complete and load-bearing structures, as well as their capacity for post-failure recovery. Experiments with six representative LLMs revealed significant gaps in deterministic validity, dynamic success, and post-failure recovery.

AnyAudio-Judge is a new AI evaluation model for audio instruction following. It uses a dynamic rubric-based approach to evaluate audio captions and can improve instruction alignment in reinforcement learning for audio generation. It outperforms current state-of-the-art baselines and provides interpretable reward signals.

This research introduces KinshipQA, a procedurally-generated benchmark for multi-hop kinship reasoning that tests large language models' (LLMs) ability to reason across different cultures and reasoning hops. The study found that LLMs struggle to adapt to culturally-marked classification, with a 40.9% accuracy drop compared to biological multi-hop reasoning. The results suggest that LLMs require additional training or rules to overcome cultural biases and improve performance.

Researchers propose a new benchmark, BUILD-BENCH, to evaluate the performance of LLM agents in compiling real-world open-source software. The benchmark includes diverse OSS projects with various characteristics, and a strong baseline LLM-based agent, OSS-BUILD-AGENT, achieves state-of-the-art performance on it. This work aims to improve the realism and complexity of evaluating LLM agents in software engineering tasks.

A new benchmark, NeuroCognition, is introduced to evaluate the cognitive abilities of large language models (LLMs). The benchmark is grounded in neuropsychological tests and targets foundational cognitive components such as abstract relational reasoning, spatial working memory, and cognitive flexibility. The evaluation reveals that LLMs perform strongly on text-based tasks but struggle with image-based tasks and increased complexity. This highlights the limitations of current LLMs and provides insights for improving their cognitive abilities.

TripScore is a benchmark and evaluation framework for language models (LLMs) in travel planning. It uses real user logs and expert judgments to assess LLMs' ability to understand and respond to user inputs, and it compares various fine-tuning approaches for LLMs in this domain.

A new benchmark, oMeBench, is introduced for evaluating large language models (LLMs) in organic mechanism elucidation and reasoning. The benchmark consists of 10,000 annotated mechanistic steps with reaction type labels, intermediate structures, and difficulty ratings. Evaluation of state-of-the-art LLMs shows that while they have promising chemical intuition, they struggle to produce consistent reasoning across multi-step mechanisms. Combining prompting strategies with fine-tuning enables smaller-scale models to achieve performance comparable to larger models.

PetriBench is a new benchmark for evaluating large language model (LLM) reasoning over dynamic state spaces using Petri nets. It assesses LLM capabilities in four task families with varying scope and temporal horizons, and its results show that accuracy decreases with difficulty and that test-time compute interacts differently with different reasoning tasks.

MTVA-Bench is a new benchmark for evaluating language models in cascaded voice agents, specifically addressing limitations of existing evaluation methods. The benchmark assesses the language model's performance in a real-world scenario, considering factors such as transcription issues, caller's voice, and script compliance. The results show significant variations in performance across six models, highlighting areas for improvement.

BioPhys-Bridge is a novel benchmark dataset for evaluating evidence-grounded scientific reasoning in biophysics research. It contains 500 cases, 1,517 tasks, and covers six biological domains and nine physical model families. The dataset enforces strict quality gates for evidence integrity and quantitative grounding, and preliminary evaluations show that certain models perform well in retrieving evidence IDs.

A new benchmark, HINTBench, is introduced for evaluating agent safety through non-attack intrinsic risk auditing. It includes 596 agent trajectories, with 400 synthetic risky and 136 synthetic safe trajectories, and 30 reconstructed real-world risky and safe trajectories. The benchmark supports three tasks: risk detection, risk-step localization, and intrinsic failure-type identification.

AMIGO is a new benchmark for evaluating agentic vision-language models, specifically their ability to ground information in a sequence of attribute-focused questions and answers. The benchmark stresses question selection, constraint tracking, and fine-grained discrimination, and provides metrics for evaluating model performance.

A new benchmark for zero-shot topic localization in historical Czech documents has been introduced. The benchmark includes a human-annotated dataset with topics and annotated spans, and evaluates the performance of large language models and BERT-based models. The results show variability among LLMs, with some approaching human agreement, making it a useful resource for testing and improving LLMs.

Researchers introduce ShotFinder, a benchmark for open-domain video shot retrieval that includes a three-stage retrieval and localization pipeline. They propose using large language models for query expansion, candidate retrieval, and description-guided shot localization. The benchmark reveals a significant gap to human performance, with challenges in color and visual style localization.

A new benchmark, WordPolo, evaluates Large Language Models (LLMs) and Large Reasoning Models (LRMs) on their reasoning processes, providing insights into their capabilities beyond accuracy alone. The benchmark requires models to navigate semantic space and systematically narrow the search, making iterative reasoning and adaptive search strategies observable and necessary for success.

ProgramDistill is a benchmark for evaluating coding agents, specifically those used in software development. It evaluates agents on features discovered through interaction with fully functional reference applications, providing a scalable and controlled difficulty for testing and training.