标签：#ollama

For about 3 months I've been running an autonomous agent — one that thinks up and writes its own social media posts and comments — unattended, 4 sessions daily, on a 16GB M1 Mac with small models in the 9B / E4B class. I'm about to publish what that operation taught me about hardening, as a series of 4 technical articles. Before that, there's one thing I want to write down first: why small models . I've been to the purchase page for a Mac Studio or a new MacBook Pro more than once or twice. Backing the agent with a large cloud model (Opus or the GPT family) has always been an option in the code. And yet I haven't bought, and I haven't switched. The 16GB M1 is not an economic constraint — it's a constraint I chose . From the outside, building on small models looks like a cheap compromise. This article explains why it isn't, and states where I stand. It also serves as the hub for the 4-article series. A model's intelligence hides the roughness of your design Large models absorb sloppy prompts, ambiguous instructions, and missing guards with sheer intelligence. If all you want is to ship a product, that's a virtue. But if you want to become someone who can build things , it becomes a defect. Because inside the thing that worked, you can no longer tell where your design ends and the model's intelligence begins. "It worked" and "I built it" are different things. Something you bludgeoned into working with model capability counts as a thing that ran — it doesn't become the ability to build. Small models have no absorption capacity. So every design flaw comes to the surface. In my operation, all of the following surfaced: The context window being silently truncated Outputs cut off midway A runaway caused by one missing sampling parameter In cloud or large-model environments, these rarely bother you. The environment has cushioning built in. Context windows are in the 200K–1M token class, so truncation itself rarely happens. And when you do exceed the limit, you get an explic

TL;DR Short version: no. I dropped a much older GPU ( Quadro P2000, 5GB, Pascal, 2016 ) next to an RTX 3090 (24GB, Ampere) on the same box, ran the same context-length ladder (8K→128K) through Ollama and vLLM on qwen3-coder:30B-A3B , and got zero extra usable context in either engine — and a 74% decode-speed hit for the trouble. Ollama hits the identical Chunk too big wall at ctx=65536 whether the P2000 is there or not. vLLM refuses tensor-parallel across the two cards entirely — not a VRAM problem, a flat compute-capability rejection ( Minimum capability: 75. Current capability: 61. ) that fails in 40 seconds, before any memory profiling. And the one real, measured effect of adding the P2000 to Ollama: decode speed goes from 76 → 19.5 tok/s at ctx=49152 once the P2000 gets pulled in as an actual compute device. Full narrative version — the two-stage collapse, the prompt-cache validation bug caught mid-sweep, the CUDA13-silently-drops-Pascal finding — is on Medium .## The setup ardi (dual Xeon E5-2680 v4, 128GB RAM, openSUSE Leap) has a Quadro P2000 sitting in a second slot next to the RTX 3090 this whole series has run on so far. Same model as phase 1 ( qwen3-coder:30B-A3B ), same box, four legs: {Ollama, vLLM} × {3090 only, 3090+P2000 tandem}, priced through HomeLab Monitor against real GPU power draw. Ollama: same wall, extra tax ctx 3090 only decode tok/s tandem decode tok/s P2000 VRAM (tandem) 8,192 124.3 122.0 6 MB / 0% 24,576 108.2 70.0 62 MB / 0% 32,768 99.4 61.0 62 MB / 0% 49,152 75.7 19.5 3,580 MB / 55% 65,536 fatal: Chunk too big fatal: identical Chunk too big — Two separate costs, not one: decode already falls behind at ctx=24576 while the P2000 is still basically idle (62MB, 0% util) — some scheduling overhead just from having a second visible device. Then the real collapse hits at ctx=49152, when the P2000 actually gets pulled into the compute path (3.58GB, 55% util) and decode craters to 19.5 tok/s . Same context ceiling either way, worse speed the wh

How I ended up running a local LLM to generate my git commit messages

The Promise: A Private, Voice-Activated Shell The dream of a voice-activated command line is compelling: speak a command, see it executed. But for many developers, piping terminal input through a cloud-based API is a non-starter. This is the promise of a project like karolswdev/HoldSpeak , a cross-platform tool for local voice typing. Could it be the core of a truly local-first, push-to-talk shell assistant? I paired it with Ollama and a local llama3.2 model to find out. The goal was simple: hold a key, speak a command like "list files by size," release the key, and have the correct shell command appear, gated by a final confirmation prompt. This project turned out to be a tale of two stacks: one for voice that was surprisingly clean, and one for language that revealed the sharp edges of the local-first promise. Building the Demo To test this idea, I built a small Python script to tie these components together. You can find the complete code for this experiment, including the prompt engineering, in my demo project on GitHub: voice-activated-shell-demo . Setup Instructions Recreating this local-first voice assistant involves a few distinct steps: Install HoldSpeak from Source : Since we need to use it as a library, clone the repository and install it in editable mode. git clone https://github.com/karolswdev/HoldSpeak.git cd HoldSpeak pip3 install -e . Install and Run Ollama : Use Homebrew (on macOS) to install the Ollama CLI, then start the server. brew install ollama ollama serve Pull a Local LLM : In a separate terminal, pull a small, capable model. I used llama3.2 . ollama pull llama3.2 Grant Permissions (macOS) : To allow the hotkey listener to work, your terminal application (e.g., iTerm, Terminal.app) must be given Accessibility permissions in System Settings > Privacy & Security > Accessibility . Run the Demo Script : With the setup complete, you can run the final Python script that integrates all these components. Finding the Seams in HoldSpeak HoldSpeak pres

Here's a thing that catches almost everyone the first week they run a model locally. You paste a 600-line file into your shiny new local assistant, ask it to find the bug, and it confidently rewrites a function that isn't even in the part it read. No error. No warning. It just... silently dropped most of your file on the floor before the model ever saw it. That's not the model being dumb. That's Ollama doing exactly what it was told. By default it gives every model a context window of 2048 tokens and quietly truncates anything past that. It's one of a handful of small surprises that separate "I installed Ollama" from "I actually understand what's running on my machine." Let's go through the ones that matter: how the thing works under the hood, what hardware you really need, the gotchas, and the honest answer to "should I even bother instead of just calling an API?" What Ollama actually is Ollama gets described as "Docker for LLMs," and that's a decent first approximation. You pull a model, you run it, there's a registry. But it hides what's doing the heavy lifting. Underneath, Ollama is a friendly wrapper around llama.cpp , the C/C++ inference engine that made running these models on consumer hardware practical in the first place. When you type ollama run , you're really booting a llama.cpp runtime with a sane default config and a tidy HTTP server bolted on. The models it runs are in a format called GGUF (GPT-Generated Unified Format). A GGUF file isn't just weights. It's a self-contained package that bundles the tensors, the tokenizer config, the architecture details, and hyperparameters like the trained context length, all in one file. That's why ollama pull llama3.1 gives you something that just works: everything the runtime needs to reconstruct the model is in the box. Ollama itself is young. The project shipped its first release in early July 2023 , and it rode the wave of open-weight models (Llama 2 landed that same month) that suddenly made "run a real LLM on

Hermes-Crew Hybrid: A Hybrid Architecture for Secure Multi-Agent AI Workflows I built a hybrid system that combines a central orchestrator (Hermes) with temporary CrewAI micro-crews, protected by 3 layers of security. Here's what it does and why it matters. The Problem Multi-agent AI systems are powerful but dangerous. When you chain multiple agents together, a single compromised agent can poison the entire workflow. Existing solutions are either too heavy (enterprise PKI infrastructure) or too light (basic regex filters). The Solution: 3-Layer Security Layer 1 — Pre-execution (MCP Tool Auditor): Before any agent can register a tool, it's audited for malicious instructions. Layer 2 — Runtime (Agent Fixer Stage): Every output from every agent passes through a 3-stage pipeline (normalization → pattern matching → embeddings) in under 1ms. Layer 3 — Pre-commit (Code Safety Hook): Before any git commit lands, the diff is analyzed by CrewAI + Ollama local. Malicious code gets rejected automatically. Architecture Hermes (Director) │ ├── MCP Tool Auditor → verifies tools before registration │ ├── Execution: venv (fast) / Docker (isolated) / auto (smart) │ ├── Agent 1: Researcher │ ├── Agent 2: Analyst │ └── Agent 3: Writer │ ├── Security Gateway (Agent Fixer Stage) → filters output (<1ms) │ └── Consolidator → parses output + generates Obsidian notes What Makes It Different 1. Portable by design. Zero hardcoded paths. Every user configures their own .env . 2. Multi-model via LiteLLM. Works with Ollama local, OpenAI, Anthropic, Gemini, Groq, OpenRouter — any provider. 3. Local-first. Everything runs on the user's machine. No cloud dependencies required. 4. Obsidian integration. Every analysis generates a structured note with YAML frontmatter. Code Safety Hook in Action When you run git commit with malicious code: ❌ [ COMMIT RECHAZADO] Code Safety detected risks: → CrewAI detected vulnerabilities: VERDICT: FAIL → Agent Fixer Stage detected anomalies: High threat score: 1.05 Fo

Last week I couldn't shake an idea: what if I had an AI that knew everything I know ? Not ChatGPT — something on my hardware, holding my knowledge, answering to no one's API bill. Yesterday I built it. Here's the honest breakdown. What it does NEXUS runs on a regular Windows laptop — aging i7, 16GB RAM, no GPU. It: Remembers everything. Drop any file in a folder; 60 seconds later it's searchable memory. Answers from MY knowledge. "Which of my projects were formally closed and why?" — it answers from my actual records. Watches the live web. Every 2 hours it pulls Hacker News and news feeds, learns what's trending, pings my Telegram. Reports to my phone. 7 AM daily briefing: what it learned, what's running, what needs me. The stack — all free, all open source Ollama runs the models (Llama 3.2, Mistral 7B). Open WebUI is my private ChatGPT. Qdrant stores memory. n8n automates. SearXNG searches privately. PostgreSQL, Redis, and MinIO handle data. Commercial equivalent: $300–500/month . My cost: electricity. The memory trick nobody explains simply Parse — extract text from any file Chunk — split into ~300-word pieces Embed — each chunk becomes 768 numbers representing its meaning Store — a database that searches by similarity Your question becomes 768 numbers too, and the database finds memories with similar meaning — not matching keywords. I asked "how do I get clients cheaper" and it found my notes on "reducing customer acquisition cost." Different words. Same meaning. That's the magic. What surprised me A 2GB model is genuinely useful. Llama 3.2 3B answers from my knowledge in seconds, on CPU. The automation matters more than the AI. The watched folder + Telegram bot turned a cool demo into a system I actually use. Windows is fine. Docker Desktop + WSL2 ran all nine services without drama. The bill, honestly Hardware: $0 (laptop I own) Software: $0 (open source) APIs: $0 (all local) Time: one focused day The only future cost is a cloud GPU server (~$65/mo) when I outg

Coding agents — Codex CLI, Claude Code, Cursor, and Pi — are productivity multipliers. But they all assume you are happy sending your code to someone else's servers. For many of us that is a deal-breaker: proprietary codebases, client NDAs, compliance requirements, or just the principle of owning your own compute. This guide shows how to swap out every cloud API with a local Ollama server running qwen3-coder:30b . Same tools, same workflows, no data leaving your network. Why Run AI Locally? The case is simple: Zero data exfiltration. Your code never leaves your machine or LAN. No per-token cost. Run 10,000 completions or 10 — the electricity bill does not care. Works offline. Airplane mode, restricted network, flaky VPN — irrelevant. No rate limits. No 429s at 2 am when you are in flow. The honest tradeoff: frontier models (Claude Opus 4, GPT-5) still outperform local models on complex multi-step reasoning and very large context tasks. For the 80% of day-to-day coding work — autocomplete, refactors, test generation, documentation — a well-chosen local model is more than good enough. Hardware Requirements I run this on an Apple M4 Pro with 48 GB unified memory . Apple Silicon's unified memory architecture is exceptionally well-suited to LLM inference: the GPU and CPU share the same memory pool, so a 22 GB model fits comfortably alongside a full development environment. Minimum viable setup: RAM What fits 16 GB 7–8B parameter models (qwen3:8b, llama3.2:8b) 32 GB 14–20B models (qwen3:14b, gpt-oss:20b) 48 GB 30–35B models (qwen3-coder:30b, qwen3.6:35b) 64 GB+ 70B models (deepseek-r1:70b, llama3.3:70b) On Intel/AMD systems with discrete GPUs the math is different: VRAM is the bottleneck, and models that don't fit entirely in VRAM fall back to slow CPU offloading. Choosing a Model For 48 GB unified memory, these are the models worth knowing about: Model Size on disk Active params Strengths qwen3-coder:30b ~22 GB 3.3B (MoE) Coding, 256K context, HumanEval SOTA qwen3.6:35b

The Problem With Choosing a Local Model Everyone has an opinion on which local LLM is best. "Use Llama — it's the most popular." "Mistral 7B has the best quality." "Phi-3 Mini is small and efficient." None of these claims come with numbers. Specifically: your numbers, on your hardware, for your workload. I built a benchmarking system to change that. Three models, 30 prompts, full latency distribution, memory profiling per inference call, and a JSON validation layer to measure structured output reliability. Here's what I found — and why the results matter for anyone deploying local models in production. The Setup Three models tested: llama3.2:3b — 3B parameters, Q4_K_M quantization, 2 GB download phi3:mini — 3.8B parameters, Q4_K_M, 2.3 GB download mistral:7b — 7B parameters, Q4_K_M, 4.1 GB download Hardware: CPU only, no GPU acceleration. This is the worst-case baseline — the scenario that exposes real latency and memory numbers. 30 test prompts across 5 categories: Short factual (10): "What is the capital of France?" Reasoning (8): "Explain why the sky appears blue." Code generation (5): "Write a Python function to reverse a string." Structured output (5): "List 3 frameworks in JSON format with name and use_case." Multi-step (2): Complex chained reasoning tasks. Architecture POST /query → Pydantic validation → Ollama HTTP API → JSON Validator → QueryResponse POST /benchmark → Load test_prompts.json → For each prompt: psutil memory before → Ollama → psutil memory after → NumPy: P50/P95/P99 latency, avg TPS, peak/avg memory → BenchmarkResult JSON The benchmark runs prompts sequentially, not in parallel. Parallel would contaminate the per-prompt memory measurements. Results Llama 3.2 3B (Q4_K_M) avg_tokens_per_second : 42.3 p50_latency_ms : 1203 p95_latency_ms : 3847 p99_latency_ms : 5120 peak_memory_mb : 6953 avg_memory_mb : 6842 total_test_duration_s : 87.4 Interpretation: P50 at 1.2 seconds is excellent. P95 at 3.8 seconds misses a 3-second SLA — the outliers are m

I recently rebuilt my homelab from scratch. The goal was simple: one machine, everything containerised, zero exposed ports, GPU-accelerated local AI, and a fully automated backup setup. No cloud subscriptions for the tools I use every day. This is the full technical breakdown — what I'm running, how it's wired together, and the hard-won fixes that cost me hours so you don't have to repeat them. What I'm Running Eight services, 26 containers, one machine: Service Purpose Portainer Docker management UI Uptime Kuma Service monitoring (7 monitors) NocoDB Self-hosted Airtable — CRM & leads n8n Workflow automation Open WebUI Local AI chat interface Ollama Local LLM inference (GPU) AFF!NE Collaborative docs & whiteboards Plane Project management (roadmaps, sprints) Duplicati Encrypted daily backups Cloudflare Tunnel Zero Trust secure access — no open router ports All external-facing services sit behind Cloudflare Zero Trust with email OTP. No passwords to manage, no VPN clients — Cloudflare handles authentication at the edge. Architecture ┌──────────────────────────────────┐ │ Cloudflare Edge (Zero Trust) │ │ *.yourdomain.com — email OTP │ └──────────────┬───────────────────┘ │ HTTPS ┌──────────────▼───────────────────┐ │ Ubuntu Machine │ │ │ │ cloudflared (outbound tunnel) │ │ │ │ │ ┌─────▼────────────────────┐ │ │ │ homelab-net (bridge) │ │ │ │ │ │ │ │ portainer uptime-kuma │ │ │ │ nocodb n8n │ │ │ │ open-webui affine │ │ │ │ plane-* duplicati │ │ │ │ ollama (GPU passthrough) │ │ │ └───────────────────────────┘ │ └───────────────────────────────────┘ Everything runs on a shared Docker bridge network ( homelab-net ). The cloudflared container maintains an outbound-only encrypted tunnel — no inbound ports open on the router at all. Ollama runs in Docker with NVIDIA GPU passthrough. The AI model inference happens on the GPU, leaving CPU headroom for all other services. Prerequisites Ubuntu 24.04 LTS Docker Engine + Compose v2 NVIDIA GPU with driver 535+ NVIDIA Container Too

This is the technical, reproducible version of a fix I shipped on my own homelab. If you want the narrative version, that's on Medium. This one is the recipe: the measurements, the math, the Modelfile, and the exact prompt I gave Claude Code to generate it. Copy-paste friendly. Repo for the dashboard used throughout: https://github.com/SikamikanikoBG/homelab-monitor TL;DR One 24GB RTX 3090, two GPU services: WhisperX large-v3 (STT, 7.7GB peak) and a Devstral Small 24B email-triage LLM (Q4_K_M, ~18.3GB). 18.3 + 7.7 = 26GB → CUDA OOM whenever they overlapped. The LLM was loaded with a 40k context window but the triage job never needed more than ~5–8k tokens. Capped num_ctx to 8192 → KV cache drops from ~6.1GB to ~1.25GB → model footprint ~18.3GB → ~14.2GB . 14.2 + 7.7 = 21.9GB → both resident, zero OOM, no quality loss. The setup Host : openSUSE, Xeon (56 threads), 125GB RAM, 1x RTX 3090 (24GB) GPU svc : WhisperX large-v3 (speech-to-text) GPU svc : Ollama -> devstral-small-2 (24B, Q4_K_M) for background email triage Both services run all the time. The OOM only happened when I dictated to my assistant (WhisperX) while the triage loop was active. Step 1 — Make the contention measurable nvidia-smi shows instantaneous VRAM. It can't show you which service spiked or when two of them overlapped — and an intermittent OOM is a timing problem. You need per-service VRAM history. I use my own dashboard (homelab-monitor) for this. The relevant view is "AI Models", which attributes VRAM per model server and per loaded model, over a time range, with OOM markers and a capacity ceiling line. What the history showed at the overlap window: Service Peak VRAM Devstral 24B (triage) ~18.3 GB WhisperX large-v3 7.7 GB Total ~26 GB on a 24 GB card If you want to reproduce the measurement, the dashboard runs as a single container: git clone https://github.com/SikamikanikoBG/homelab-monitor cd homelab-monitor docker compose up -d --build # open http://<host>:9800 -> AI Models / GPU views (NVIDI

This is the concrete, runnable walkthrough for Post 1 of the Portway series . The goal: stand up a single model behind an OpenAI-compatible endpoint on hardware you already own, call it from the official OpenAI SDK, and internalize the stateless contract. Everything here runs locally for $0. What this post covers A demo.py script with two blocks: Round-trip — one chat call via the OpenAI SDK, printing the content and the usage object. Stateless proof — the same final question sent as a 1-turn message and as the last turn of a 5-turn fabricated history; both prompt_tokens values are printed alongside an explanation of the delta. Engine choice on this machine Apple Silicon Mac, 48 GB unified memory, Ollama already installed. The demo uses Ollama's OpenAI-compatible endpoint at http://localhost:11434/v1 and the gpt-oss:20b model (~14 GB). The wider Portway series uses llama.cpp on Mac (Ollama is called out as problematic for Qwen3.5 in Post 2). For Post 1 — one model, prove the contract — Ollama is fine and already on the box. Model options by available RAM The demo script works with any Ollama-served model — just substitute the model name in demo.py . The table below covers machines from 9 GB unified memory upward. Model Pull command Approx size Min RAM Notes llama3.2:3b ollama pull llama3.2:3b ~2 GB 8 GB Fastest; good for testing the contract gemma3:4b ollama pull gemma3:4b ~3 GB 8 GB Google; solid instruction-following mistral:7b ollama pull mistral:7b ~4.1 GB 8 GB Classic 7B baseline llama3.1:8b ollama pull llama3.1:8b ~4.7 GB 9 GB Best quality under 10 GB qwen2.5:7b ollama pull qwen2.5:7b ~4.4 GB 9 GB Strong at instruction + reasoning gpt-oss:20b ollama pull gpt-oss:20b ~14 GB 24 GB Used in this post's sample output On a 9 GB machine, replace gpt-oss:20b in demo.py with llama3.1:8b or qwen2.5:7b — the contract demonstration is identical. Prerequisites Ollama running locally ( curl -s http://localhost:11434/api/tags should return JSON) uv installed ( uv --version )