Cancellation is not enough: retries can prevent recovery after transient overload in an LLM engine, tests show using vLLM with open models and scheduler traces
Read the original at arxiv.org→arXiv:2611.08684v1 Announce Type: new Abstract: Clients and proxies in front of an LLM engine retry calls that time out. After a transient overload, these retries can keep the engine from recovering even when calls...
Original headline: "Cancellation Is Not Enough: What Retries Leave Behind in an Overloaded LLM Engine"
Coverage timeline
- Nov 9, 05:00 UTC arXiv cs.AI lead source Cancellation Is Not Enough: What Retries Leave Behind in an Overloaded LLM Engine