SpecFit is a proposed approach to optimize the inference of mixture-of-experts (MoE) models by choosing the verified draft subtree to maximize predicted throughput. This can lead to significant speed improvements, up to 21.6% and 7.0% faster than EAGLE-3, and is implemented in a publicly available codebase.

ReplaySSM is a technique for improving the performance of state-space models by separating output computation from checkpoint updates, reducing state memory traffic and enabling low-precision checkpointing without affecting recurrent updates. It accelerates decoding by 1.2-1.47x and increases throughput by 1.8x.

Researchers introduce PRIME, a framework for optimizing index-side representation of documents for LLM agents, leading to improved retrieval quality and reduced computational costs.

This paper introduces BudgetAPO, a prompt optimizer that adapts to tight budgets by using noise-adaptive evaluation and a reflective operator. It outperforms existing methods like GEPA and OPRO on seven benchmarks and five subject models.

Researchers developed a memory layer for large language models, galahad-kv, which saves and loads model state without recomputing it, reducing GPU energy consumption and increasing performance.