Stable Token-level Scheduling for Serverless LLM Inference
Researchers propose Token Window Pacing (TWP), a closed-loop scheduling framework for serverless Large Language Model (LLM) inference that regulates request admission at the token level to minimize latency and improve throughput.
Save an API key to vote.