Latency-aware LLM query routing for dynamic workloads improves by considering generation latency alongside accuracy and cost
Read the original at arxiv.org→arXiv:2607.18253v1 Announce Type: new Abstract: Modern language query routers improve inference efficiency by assigning each query to a model that balances response quality and monetary cost. However, current query...
Original headline: "Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads"