Olla v0.1.0 releases as a lightweight open-source LLM proxy for homelabs and on-prem inference, with failover, sticky sessions, and load balancing (Ollama, vLLM, SGLang, llama.cpp, oMLX).
Read the original at www.reddit.com→Hey folks, About 18 months ago we started building Olla to solve some of the headaches we kept running into whilst managing multiple local LLM inference servers, both in our homelabs and at work. We've shared it here...
Original headline: "Olla v0.1.0 - Lightweight open-source LLM proxy for homelabs & on-prem inference (Ollama, vLLM, SGLang, llama.cpp, oMLX with failover, sticky sessions & load balancing)"
Coverage timeline
- Nov 3, 11:13 UTC r/LocalLLaMA lead source Olla v0.1.0 - Lightweight open-source LLM proxy for homelabs & on-prem inference (Ollama, vLLM, SGLang, llama.cpp, oMLX with failover, sticky sessions & load balancing)