LIVE · refreshes every 20 min

updated Oct 19, 15:23 UTC

110010.art / topic

Llama

Oct 19

2h ago Alternative approaches to base llama.cpp for RDNA2 GPUs in local LLM usage; user notes WMMA requires RDNA3+ and seeks broader RDNA2 support r/LocalLLaMA → story

2h ago Strata with multi-GPU explanation r/LocalLLaMA → story

4h ago Llama server default port changed from 8080 to 9931 (ggml) inside the container starting v0.7.0 r/LocalLLaMA → story

11h ago SonicBoom v0.0.3 outperforms llama.cpp in decoding on a desktop RTX 3050 8 GB, with benchmark results shown for Gemma 4 E4B Q3_K_M and Qwen2.5-0.5B. r/LocalLLaMA → story

Oct 18

1d ago New quant types suggested: IQ2_K, IQ3_K, IQ4_K, IQ5_K and IQ6_K r/LocalLLaMA → story

Oct 17

2d ago CUDA shows big speedup for low-bit MMVQ types like Q2_K and Q3_K in llama.cpp pull request 30517 r/LocalLLaMA → story

2d ago Kolibri-1 (Aleph Alpha) support added to llama.cpp with European 78B A3.4B model; GGML community pulls request 30673 r/LocalLLaMA → story

2d ago Llama.cpp has come a long way r/LocalLLaMA → story

Oct 16

3d ago Reflection releases Beam-1.0 with a 501B-parameter sparse MoE model (23B active) supporting 1M-token context; instruction-tuned in BF16, FP8, and NVFP4 across English-focused deployment. r/LocalLLaMA → story

3d ago MeddeX: a Retrieval-Augmented LLaMA-3 system with Gemini Vision for evidence-backed medical report analysis arXiv cs.CL → story

Oct 15

3d ago Qwen3.8FN vLLM vs. llama.cpp vs Ninfer6000 compared; Decode demonstrates strong performance with Prefill trails vLLM, close to the reported 400 tokens/s in ideal conditions r/LocalLLaMA → story

3d ago QUASAR 4-bit QAT checkpoints for Qwen3.5, Gemma 4 and Muse Glimmer; closer to BF16 than Google’s and NVIDIA’s 4-bit releases r/LocalLLaMA → story

3d ago Nvidia+AMD RPC offload increases context to 195k on 2x 4060 Tensor Parallel with no perf cost r/LocalLLaMA → story

4d ago Add Nemotron Super 3.5 VL and Nemotron Bolt 74B A9B support in llama.cpp; includes mention of weights for 74B A9B and 120B A12B citations r/LocalLLaMA → story

4d ago Titan-Llama: retrofitting pre-trained LLMs with external neural memory modules arXiv cs.CL → story

Oct 14

4d ago Qwen3.8-Next on Strata and Qwen3.8-27B on llama.cpp evaluated on IQ3_S and UD-Q4_K_XL benchmarks; results show Next and 27B performance across multiple tasks r/LocalLLaMA → story

5d ago ggml/llama.cpp adds --spec-draft-n-ubatch option to optimize VRAM usage r/LocalLLaMA → story

5d ago I built a cheap dual 3090 rig from used parts; it isn’t much faster than expected r/LocalLLaMA → story

5d ago Moving a running model’s layers between GPUs mid-reply (8 ms, no re-prefill) r/LocalLLaMA → story

Oct 13

5d ago Real downsides of Strata? downsides debated; user reports looping issue and port misunderstandings while comparing to Qwen 27B 3.8 Q4 UD and Swift Q3 Next r/LocalLLaMA → story

5d ago Why AnythingLLM is moving from Ollama to Lemonade and llama.cpp r/LocalLLaMA → story

6d ago MoE experts loaded from disk in llama.cpp's WebGPU backend in a Chrome tab for an 80B Qwen3-Next model (48 GB GGUF) r/LocalLLaMA → story

6d ago SonicBoom achieves 52 tok/s decode on a laptop RTX 3050 (llama.cpp 45) with the same quantized kernels; the difference is a graph compiler, with no Python involved. r/LocalLLaMA → story

6d ago Running decision models: picking quantization and VRAM usage r/LocalLLaMA → story

6d ago Tesla V100 SXM2 shows N/A power readings and Xid 44 errors in dmesg on Ubuntu 26.04 with NVIDIA drivers; user reports failure in llama-cli on PCIe with CUDA 12/13 r/LocalLLaMA → story

6d ago Two Android phones cluster to run a 27B LLM locally, bypassing RAM limits with C++ RPC split and Hadamard Ternary math r/LocalLLaMA → story

6d ago Local models could start optimizing their own runtime stacks; open-source parity and recent forks suggest progress next year, with Claude and OpenAI-scale models informing developments r/LocalLLaMA → story

Oct 12

6d ago How to get more performance from dual A100 80GB? r/LocalLLaMA → story

6d ago Llama.cpp optimizations for specific hardware and models r/LocalLLaMA → story

7d ago Why MTP is slower than standard decode r/LocalLLaMA → story

7d ago BrokenML: can agents fix broken ML jobs? r/LocalLLaMA → story

7d ago llama.cpp Prism Bonsai 2 27B runtime support is merged; users can run Prism Bonsai 2 27B with the original llama.cpp r/LocalLLaMA → story

7d ago Strix-llama new branch rdna35-perf boosts prompt processing 21–54% vs Vulkan and up to 2.7× vs stock ROCm kernel at 128k outputs quality unchanged r/LocalLLaMA → story

7d ago Do you actually test open-model inference compatibility? r/LocalLLaMA → story

Oct 11

7d ago Gemma4 E4B EXL3 offers an alternative to llama.cpp on Jetson Orin, outperforming llama.cpp on Jetson Orin Nano with improved prefill and lower voice-chat latency. r/LocalLLaMA → story

7d ago Every model that can be run on 10-16GB VRAM ranked r/LocalLLaMA → story

8d ago OpenMed 3.0 ships; Apache-2.0 medical AI runs fully local with no cloud calls, 422 open issues for 3.1 r/LocalLLaMA → story

8d ago Now you can grow Bonsai on your potato r/LocalLLaMA → story

8d ago Llama.cpp update adds probabilistic MTP with improvements in prose generation; draft-n-max and draft-p-min align with greedy sampling, per merge of PR 27694 r/LocalLLaMA → story

8d ago Local semantic file search for Linux uses EmbeddingGemma 2 with Rust, llama.cpp, and in-process embedding across text, images, video frames, and audio windows; hybrid ranking combines vector similarity with SQLite FTS5 r/LocalLLaMA → story

← 1 2 3 →