LIVE · refreshes every 20 min

updated Oct 19, 16:24 UTC

110010.art / topic

Llama

Oct 11

8d ago Local semantic file search for Linux uses EmbeddingGemma 2 with Rust, llama.cpp, and in-process embedding across text, images, video frames, and audio windows; hybrid ranking combines vector similarity with SQLite FTS5 r/LocalLLaMA → story

Oct 10

8d ago [Model] Support MiniCPM-V 4.7 by tc-mb in ggml-org/llama.cpp pull request #29416 r/LocalLLaMA → story

8d ago Qwen 3.8 models tested on AtCoder problems across four thinking efforts r/LocalLLaMA → story

9d ago Is there a better option than llama.cpp for 4GB VRAM to increase tokens per second without PyTorch dependencies? r/LocalLLaMA → story

9d ago Improve token per second (tps) to 40 without changing quant settings; user reports 30tps on RX 6700 XT with llama.cpp Vulkan, seeking performance tweaks while keeping q8 and q4_k_xl unchanged r/LocalLLaMA → story

9d ago Fully local conversational AI runs on a plush toy using Whisper, Hermes 8B, Kokoro; no cloud, on LAN with face identity and emotion recognition r/LocalLLaMA → story

9d ago Asus ESC4000 G3 with 8× T4 GPUs: fastest way to deploy Qwen3.5-9B for 10–15 concurrent users (current setup uses llama.cpp) r/LocalLLaMA → story

Oct 9

10d ago Mellum2.1-12B-A2.5B tested on PI Coding Agent; one-shot results mixed across tasks r/LocalLLaMA → story

Oct 8

10d ago GPU - Vulkan llama.cpp benchmarks sorted by price to performance r/LocalLLaMA → story

10d ago $2800 rig with 8x Radeon Pro V620 (256 GB VRAM) + custom vLLM fork achieves Qwen3.8-Flash-Next at 60–100 t/s decode and 3000+ t/s prefill r/LocalLLaMA → story

11d ago Llama.cpp-vulkan: best strategy to use iGPU alongside dGPU(s) without slowing performance r/LocalLLaMA → story

11d ago Java vllm-like framework achieves ~90% of llama.cpp performance for local inference on NVIDIA GPUs by compiling Java to CUDA and cuTile r/LocalLLaMA → story

11d ago Best current R9700 inference engine? r/LocalLLaMA → story

11d ago 2 x dgx spark mtp files start command stalls when launching llama-server with Qwen3.8-Flash-Next GGUF on unsloth; user reports stall in cluster setup r/LocalLLaMA → story

11d ago AI development on Windows: PyTorch and Llama.cpp to Windows ML Hacker News (AI) → story

11d ago Cloudflare Clef Experience uses llama.cpp 0.6.0 and bartowski's Q4_K_M quant for refund inquiry handling r/LocalLLaMA → story

11d ago How fragile is on-device language model safety? Localizing safety-critical parameters for sparse fault analysis arXiv cs.AI → story

Oct 7

11d ago Llama.cpp on the stage r/LocalLLaMA → story

11d ago Which is a better acronym for hardware-optimized model inference engines like HOMIE and MADE r/LocalLLaMA → story

12d ago ggml-org/llama.cpp adds GLM5Next MTP support and optimizes local usage with GLM 5 Flash MTP r/LocalLLaMA → story

Oct 6

12d ago Local AI ecosystem overview r/LocalLLaMA → story

12d ago ggml/llama.cpp adds K2 Horizon dense and MoVA support via a new PR (#29535) r/LocalLLaMA → story

13d ago ggml-llama.cpp adds -sm tensor for tensor parallelism across multiple computers r/LocalLLaMA → story

13d ago Gemma 4 26B-A4B and a 37 GB Qwen3.6 MoE run in a browser tab on a 24 GB Mac; experts streamed from disk, output matches llama.cpp r/LocalLLaMA → story

13d ago Overclocking DDR5 yields modest gains for MoE prefill and decode in llama.cpp with SYCL on Intel B70 32GB r/LocalLLaMA → story

13d ago Unsloth updates Qwen3.8-Flash-Next-GGUF to work with llama.cpp r/LocalLLaMA → story

Oct 5

13d ago llama.cpp v0.6.0 released with MTP speculative decoding for Qwen4Exp and more r/LocalLLaMA → story

13d ago SkyIsNotGreen releases Scion-35B-A3B: a llama.cpp fork with Prism-ml binaries and a draft MTP model on Hugging Face r/LocalLLaMA → story

14d ago Ok how to actually learn vLLM? r/LocalLLaMA → story

14d ago Infermeld Linux kit runs a single GGUF across an AMD GPU and an NVIDIA GPU using llama.cpp r/LocalLLaMA → story

Oct 4

14d ago GLM-4.7 benchmark compares MXFP4, Q4_K_M, and Q4_K_XL on a Radeon 680M iGPU with an AMD Ryzen 7 6800H system using llama.cpp Ubuntu Vulkan binary r/LocalLLaMA → story

14d ago QFN llama.cpp: user asks for config tips for running a squished ISTA model on dual RTX 5060 Ti and 32 GB RAM r/LocalLLaMA → story

14d ago LLM inference dashboard offers local, scalable logging and metrics across multiple engines; not public yet but seeks interest r/LocalLLaMA → story

14d ago Poorman inference engine enables 35B Qwen 3.6 to run MoE expansion with 8 routed experts per token on a 16GB GPU; Claude Code compatibility via OpenAI and Anthropic APIs r/LocalLLaMA → story

14d ago Qwen3.5 arch implemented in FPGA fabric for 9B/27B INT4 models on inexpensive eBay mining hardware r/LocalLLaMA → story

15d ago Home GPU bottleneck delays as user scales from 1x3090 to 20 DGX Sparks to run large language models locally; reports first bottleneck in house fuses r/LocalLLaMA → story

15d ago Improving creative writing for Local Models (Qwen) r/LocalLLaMA → story

Oct 3

15d ago Rise of overfit inference engines with narrow runtimes like Strata, ninfer, DwarfStar, Splash, llamAmpere, gufo, and similar trends toward specialized performance over generality r/LocalLLaMA → story

16d ago Strata demonstrates 5-10x prefill and 3-4x decode speed improvements in llama.cpp with MoE caching; llama.cpp previously showed little progress, arXiv papers cited but no implementation r/LocalLLaMA → story

16d ago Anyworld is a self-hosted multiplayer text RPG that uses a local LLM as the Dungeon Master, with browser-based play and optional cloud API support. r/LocalLLaMA → story