标签：#gguf

What flipped in b9437 Build b9437 , published on May 30, 2026 at 20:56 UTC , ships two targeted default-value corrections to llama-bench . Flash attention ( -fa ) shifts from a hard-coded off to auto ( LLAMA_FLASH_ATTN_TYPE_AUTO ), and the GPU-layer count ( -ngl ) changes from the legacy sentinel 99 to -1 . Both values now match what llama-server and llama-cli already used — the bench tool was simply never updated to track them until this build. Quick Answer: Before b9437 (published May 30, 2026) , llama-bench hard-coded -fa off , silently skipping flash attention even on CUDA, Metal, and Vulkan hardware. Build b9437 sets the default to -fa auto and -ngl -1 , matching llama-server and llama-cli . Any pre-b9437 baseline on FA-capable hardware needs a flag-matched re-run to remain valid. PR #23714 , reviewed and merged by maintainers JohannesGaessler and pwilkin, adds the same -fa auto|off|on tri-state flag to llama-bench that the rest of the toolchain already supported. With LLAMA_FLASH_ATTN_TYPE_AUTO as the new default, flash attention activates automatically when the runtime detects a capable backend (CUDA, Metal, Vulkan); on CPU-only hosts it stays off with no error and no output change. Parameter Before b9437 After b9437 Behavioral impact -fa off (hard-coded) auto ( LLAMA_FLASH_ATTN_TYPE_AUTO ) GPU-capable hosts bench with FA active by default; pre/post comparisons require explicit flag-matching -ngl 99 (offload-all sentinel) -1 (runtime decides) CPU-only builds no longer attempt full GPU offload; eliminates spurious CUDA errors when no GPU is present The following verified script (executed successfully, exit 0) demonstrates the behavioral gap in concrete terms — on a capable GPU, the pre-b9437 defaults schedule zero FA rows while b9437 defaults schedule one: def old_llama_bench ( device ): # Before b9437, the default bench matrix used FA=0, so FA rows were skipped. return [{ " device " : device [ " name " ], " ngl " : 0 , " fa " : 0 }] def b9437_llama_bench ( de

NVIDIA's Parakeet speech models used to mean a Python stack: NeMo, PyTorch, and a GPU you kept warm. A new C++ port collapses that to one binary and one file. From NeMo to GGUF: What the Port Covers parakeet.cpp is a C++17 inference port that runs NVIDIA's Parakeet automatic speech recognition (ASR) models on the ggml tensor library — the same engine behind whisper.cpp and llama.cpp — with no Python, no NeMo, and no ONNX at inference time. The project is maintained by Ettore Di Giacinto (@mudler), author of LocalAI, and its first tagged release, v0.1.0, landed on May 30, 2026 . The code is MIT-licensed; the model weights keep their original NVIDIA Parakeet licenses. This is a community project, not an official NVIDIA release. The port covers the offline Parakeet families — CTC, RNNT, TDT and hybrid TDT-CTC — in 110M, 0.6B and 1.1B sizes, plus a streaming 120M model with end-of-utterance detection . Two checkpoints anchor most use: parakeet-tdt-0.6b-v2 , the English default that reports 6.05% average WER on the Hugging Face Open ASR Leaderboard and was released 05/01/2025 , and parakeet-tdt-0.6b-v3 , which extends the same 600M FastConformer-TDT architecture to 25 European languages with automatic language detection, released 08/14/2025 . Inference runs on CPU, CUDA, HIP (AMD ROCm), Vulkan and Metal (Apple Silicon) — the same ggml backend matrix as whisper.cpp and llama.cpp — so deployment reduces to one binary plus one GGUF file . The hard part was mapping Parakeet's RNNT/TDT decoders onto a static-graph tensor library. An earlier work-in-progress port surfaced on Hacker News in mid-2025, where the author flagged how far there was to go: "The GGML build is roughly 1000x slower than the MLX Python version" — jason-ni, reporting an early Parakeet-on-ggml experiment (source: Hacker News, 2025 ; see also the jason-ni port ). The mudler release is the matured answer to that decoder-on-static-graph problem, and as of June 2026 Parakeet is not yet merged into mainline whis