Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation
Researchers introduced Dooly, a profiling tool for Large Language Model (LLM) inference simulation. Dooly improves on existing profiling methods by being configuration-agnostic and redundancy-aware, allowing it to reuse knowledge across different model configurations. This is achieved through a single inference pass, taint propagation to identify input dimension origins, and selective profiling. Dooly reduces profiling GPU-hours by 56.4% compared to existing methods while maintaining high simulation accuracy within 5% Mean Absolute Percentage Error (MAPE) for Through-The-Firewall-Time (TTFT) and 8% for Throughput-Optimized-Time (TPOT).
Save an API key to vote.