SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference
SpecFit is a proposed approach to optimize the inference of mixture-of-experts (MoE) models by choosing the verified draft subtree to maximize predicted throughput. This can lead to significant speed improvements, up to 21.6% and 7.0% faster than EAGLE-3, and is implemented in a publicly available codebase.
Save an API key to vote.