标签：#homelab

The failures that cost me the most in three years of self-hosting were never the ones that threw an error. An error is a gift: it tells you where to look. The expensive ones are the failures that report success while being broken . A page that returns 200 OK . A healthcheck that says the container is fine. A backup that exits cleanly. A command that prints nothing wrong. Everything green, everything lying. Here are four of them, all from the same box (a 2016 desktop, i7-6700 / 32 GB, Docker behind Caddy, reachable only over Tailscale). Each fails by handing you a success signal. Each cost me an evening the first time. The fixes are boring once you know them, the point is knowing the failure exists. Sanitized skeleton with all the config at the end. 1. A loading page that returns 200 This one I could find nothing written about, so it cost me the most. To keep the box quiet, I run the heavy services on-demand: Sablier stops idle containers and starts them on the first request. Caddy (with the Sablier plugin) gates a virtual host behind a container group, serves a "please wait, starting up" page while the group boots, then proxies through: myhost . my - tailnet . ts . net : 8081 { route { sablier http :// sablier : 10000 { group office session_duration 30 m } reverse_proxy nextcloud : 80 } } I gate my whole Nextcloud vhost, WebDAV included, this way. And here is the silent failure: if the gated group is not healthy, Sablier serves that HTML loading page for every request, and it serves it with 200 OK . A browser shows a spinner, fine. But my Obsidian vault syncs over WebDAV, and a WebDAV client asking for a directory listing got a 200 with a chunk of HTML instead of the XML it expected. Sync died with a cryptic no root multistatus found . Nextcloud itself was up and perfectly healthy the whole time. Every uptime check I had was green, because the gate in front kept answering 200 . The structural lesson: the moment you put a service on-demand behind a reverse proxy, tha

Originally published at woitzik.dev A previous article here covered setting up gitleaks for homelab secret scanning - the setup, the pre-commit hook, getting CI to fail on new commits that contain secrets. The setup was correct. The tool was running. The CI was green. And it had been quietly suppressing a live production credential for months. This is the follow-on story: not about getting gitleaks running, but about the specific way a baseline file breaks the guarantees you think you have once it's in place. View the complete homelab infrastructure source on GitHub 🐙 What a Baseline File Does (and Is Supposed to Do) When gitleaks first runs on an existing repo, it finds every secret-shaped string in the full git history - including secrets that were introduced years ago, rotated long since, and are completely inert. Flagging those in CI creates noise that causes developers to tune out gitleaks entirely, which is worse than not having it. The baseline workflow is the standard answer: run gitleaks on the current state, export all findings to a JSON file, commit that file to the repo, and tell gitleaks to suppress any finding that already appears in the baseline. Future commits that introduce new secrets still fail; old known-inert findings don't. # Generate baseline from current HEAD gitleaks detect --report-format json --report-path .gitleaks-baseline.json # Tell gitleaks to use it gitleaks detect --baseline-path .gitleaks-baseline.json The assumption embedded in this workflow: findings that appear in the baseline are inert. They were there before the baseline was generated; they've been there; they're known. The Assumption That Broke It The baseline was generated at a point when the repo contained Garage's rpc_secret and admin_token committed in a YAML file. Those were real production values - the cluster was live, using those exact secrets - but the baseline suppression treated them as "known, reviewed, not a problem." The commit that introduced them had happened

The homelab started flat. One /24, everything on it. My workstation, the NAS, the Proxmox host, and — over time — a growing list of workloads sharing the same broadcast domain because that was the path of least resistance. For a while, that was fine. A homelab running one workload doesn't need segmentation any more than a house needs an office door. Then I stood up an Akash provider. An Akash provider is, in shape, a Kubernetes cluster that accepts inbound tenant workloads from the internet — real deployments, paying for compute, containers I didn't write landing in namespaces on my hardware. The provider itself is documented at github.com/jjozzietech/akash-provider-ops-public — this piece is about the network underneath it. The containerisation posture itself is fine. I trust the isolation model. But trust isn't a network design. And the network at that moment had the tenant workload cluster sitting on the same subnet as my workstation, my NAS, and my Proxmox management interface. That was the moment I stopped thinking of the rack as a home network with extra boxes, and started thinking of it as a small data centre. This piece is the network design that came out of that shift. I'll cover the layout, the rules that hold it together, and the Nexus and Proxmox configs that anchor it — with the specifics of my own deployment sanitised. It's not a step-by-step replication guide. It's the design pattern, with enough of the shape to be useful and enough restraint to not double as a recon document for my own rack. // the original design The flat layout looked like this: home lan — 192.168.1.0/24 opnsense (perimeter) cisco nexus (dumb L2 switching) proxmox host workload VMs (all on the same subnet) What it got right: zero routing complexity, everything reachable from everywhere, fast to stand up. If you're running one project on a homelab, this is the correct design. Don't over-engineer it. What stopped working, as soon as the second project landed on the rack, was that the

I gave GLM-4.5-Air (106B, open weights) 12 coding tasks through opencode on my RTX 3090. It scored 0% — never edited a single file. Same model, same GPU, same tasks, but driven by a ~150-line LangGraph agent instead: 93% . The model was never the problem. The orchestrator was. Here's the benchmark — including the part nobody else measures, the electricity cost per correct task . Setup RTX 3090 (24 GB) + 128 GB RAM , models via ollama , Q4 quants, temp 0.2 5 recent open models × 2 orchestrators (opencode vs custom LangGraph ReAct with ollama-native tool-calling) 17 graded tasks (12 coding in Python/JS/C++ + 5 general-agent) with hidden unit tests Every run priced in GPU watts via my open-source homelab-monitor Results Model tok/s opencode adh. LangGraph adh. LangGraph coding LangGraph general Qwen3-Coder 30B-A3B 130 92% 100% 100% 100% GLM-4.5-Air 106B 5.7 0% 100% 89% 100% Devstral Small 24B 49 8% 53% 8% 40% Seed-OSS 36B 9.5 0% 7% 0% 20% DeepSeek-R1-Distill 32B 6.7 0% 0% 0% 0% Tool-adherence = % of tasks where the model actually called a tool instead of just printing code in chat. It was the master variable. (GLM's headline "93%" is its blended score across all 17 tasks: 89% coding + 100% general.) Three takeaways The framework can matter more than the model. opencode sends a frontier-shaped system prompt + 12 tools over its OpenAI-compat path; most local models fall back to chatting. Native tool-calling through a lean agent fixes that — GLM went 0% → 93%. (Qwen3-Coder is the exception: it's tuned for agentic tool use and aces opencode out of the box.) Acting ≠ solving. LangGraph made Devstral act (8% → 53% adherence) but not solve (coding stayed 8%). The framework decides whether a model acts; the model decides whether it's right. The wattmeter ranks honestly. Qwen solved tasks at ~0.0005 BGN each; the models that scored zero still burned 10–30× more energy for nothing. On a home rig, the cheapest model is the fast, correct one — and MoE (Qwen activates ~3B of 30B pe

"It runs on my own GPU, so it's basically free." I believed that until I put a meter on it. So I ran a controlled benchmark on one box — an openSUSE machine with a single RTX 3090 — driving three local models through ollama under an identical fixed workload (256-token generations in a loop for ~4 minutes each), while my open-source dashboard priced every run by the real GPU energy it burned : power sampled from nvidia-smi every 10 s, integrated over each run's exact window, multiplied by my actual day/night tariff. One number per model, in euros per million output tokens. Here's the part that made me re-run it. The tiny gemma3:1b came out at €0.118 / 1M tokens — about 5× cheaper than a hosted Flash-class API (~€0.55). But gemma3:27b 's electricity alone was €0.706 / 1M — more expensive per token than just paying the cloud, and that's before a single cent of the GPU's purchase price. "Local" didn't make it cheaper; it made it cost more and I own the depreciation. The mechanism is one line: each token costs watts ÷ throughput , and a big dense model is both slow and thirsty. A newer mid-size architecture ( gemma4:26b ) bought a lot of that back, landing at €0.272 . The full guide is methodology-first and reproducible end to end — minting an ingest key, the stdlib-only client, the exact ollama loop that reads eval_count / eval_duration for real tokens-per-second, reading each run back priced, and the honest caveats (this is marginal GPU energy only — not capex, idle, or cooling — and the absolute numbers round to fractions of a cent; the shape is the finding). Read the full guide on Medium → https://medium.com/@arsen.apostolov/how-much-does-it-actually-cost-to-run-a-local-llm-per-million-tokens-measured-4a90a7f31a48

Originally published at woitzik.dev I assumed my homelab repo was clean. No one had ever flagged anything in review (there is no one else reviewing it), CI was green, and I generally try to use Vault and ExternalSecrets for anything sensitive. Then I ran a full-history gitleaks detect against it. It found 12 distinct secrets committed in plaintext — including the OIDC private key that signs SSO tokens for half the cluster. This is the scanning setup I put in place afterward, the baseline strategy that let me adopt secret scanning without getting blocked by my own history on every commit, and the remediation plan for the leaks themselves. View the complete homelab infrastructure source on GitHub 🐙 What Gitleaks Found gitleaks detect --no-banner -v Twelve real findings, plus one already-hashed password (lower severity but still shouldn't be hand-committed) and one false positive in ROADMAP.md (documentation text that happened to match a generic API key pattern). The real findings, by severity: File Secret Why It Matters kubernetes/apps/authelia/configmap.yml OIDC issuer private key Signs SSO tokens for ArgoCD, Vault, Grafana — highest blast radius kubernetes/apps/garage/config.yml RPC secret + admin token Storage backend for Velero/Loki/CNPG backups kubernetes/apps/garage/secrets.yml Admin token (duplicate) Same secret committed twice in two files terraform/stacks/network/local_backend.hcl Garage S3 access key This is the Terraform state backend's own credential kubernetes/system/postgres/cnpg-backup-secret.yml Garage S3 secret key Used for WAL archiving kubernetes/apps/paperless/secrets.yml Postgres password + AI API token kubernetes/apps/cloudflared/secrets.yml Cloudflare Tunnel token kubernetes/apps/headscale/config.yml OIDC client secret Must match Authelia's client config kubernetes/system/monitoring/loki.yml Minio/S3 password kubernetes/apps/mikrodash/secrets.yml Dashboard password Lowest priority — internal tool only None of these were exposed by a public repo

How I replaced Google Photos, Google Drive, and Google Home with a self-hosted Raspberry Pi 5 setup - and what the full technical stack looks like. There's a quiet moment every tech-literate person eventually hits. You open your cloud storage dashboard, see the number creeping up, and think: I'm paying a subscription fee, every month, forever, just to store my own photos and files on someone else's computer. That was me, but with a bit of added weight. It's not just my data I'm responsible for. Over the years, I've quietly become the unofficial digital curator for my entire family . The person everyone sends photos to after a birthday party. The one who backs up the wedding videos. The one who scans and stores the important documents, passports, contracts, and sentimental things so they don't get lost. My parents, siblings, extended family: if something matters and it's digital, there's a good chance it eventually lands with me. That's a responsibility I take seriously. And for a long time, Google was my answer - until the storage bill started quietly creeping past 2TB, and I started thinking more carefully about what it actually means to hand all of that data over to "big tech" . So I built my own home server. A tiny, almost silent box sitting in my house that now handles everything Google Drive and Google Photos did, plus more, for a one-time hardware cost of £340. This is the story of how I did it, why I did it, and exactly how you can too. Why I Did It The cost was the trigger, but it wasn't the only reason. When you use Google Photos, Google Drive, or any cloud storage service, your files live on their servers. You're trusting a corporation to keep them safe, not snoop through them, not change their pricing, and not shut down the product one day. Google has a long history of killing beloved products. Google Photos itself famously ended its unlimited free tier in 2021. As well as this, recent AI trends and auto opt-in data processing had me thinking there had to

I run a transparent OPNsense bridge between a UniFi Dream Machine Pro and the rest of my LAN. It is deliberately boring at Layer 3: the UDM keeps routing, DHCP, DNS, firewall policy, WAN handling, and VLAN definitions. OPNsense sits inline as a Layer 2 bump in the wire. The interesting part is that both sides of that bump use LACP . I already wrote the build/configuration guide for this setup here: Building a Transparent LAGG (LACP) Bridge with OPNsense, UDM, and UniFi - A Practical Guide . That article explains how the bridge was built, how the LAGG devices were configured, and why I wanted the firewall to remain transparent. This article is the other half of the story: what happens when that kind of setup fails in a non-obvious way. Not a clean outage. Not a single "the network is down" moment. Just enough instability to make everything feel wrong. 1. Topology and Failure Surface The topology looked like this: +----------------------+ | UniFi Dream Machine | | kantharos-udm-pro | +----------+-----------+ | LACP aggregate, 2 x 1G | OPNsense lagg0 "ingresslagg" igc1 + igc2, LACP | +----------v-----------+ | OPNsense bridge0 | | "laggbridge" | +----------+-----------+ | OPNsense lagg1 "egresslagg" igc4 + igc5, LACP | LACP aggregate, 2 x 1G | +----------v-----------+ | UniFi USW-Lite-16 | | downstream LAN | +----------------------+ On OPNsense, the relevant interfaces were: igc1 + igc2 -> lagg0 -> ingresslagg -> toward UDM igc4 + igc5 -> lagg1 -> egresslagg -> toward USW lagg0 + lagg1 -> bridge0 -> laggbridge The bridge is a FreeBSD bridge. The aggregates are FreeBSD lagg(4) interfaces using LACP. OPNsense exposes those through its Interfaces > Devices UI. The expected healthy OPNsense state is: laggproto lacp status: active laggport: igcX flags=<ACTIVE,COLLECTING,DISTRIBUTING> laggport: igcY flags=<ACTIVE,COLLECTING,DISTRIBUTING> Those three member states matter: ACTIVE : the member is participating in the LACP bundle. COLLECTING : the member may receive traffic. DIS

This is the technical, reproducible version of a fix I shipped on my own homelab. If you want the narrative version, that's on Medium. This one is the recipe: the measurements, the math, the Modelfile, and the exact prompt I gave Claude Code to generate it. Copy-paste friendly. Repo for the dashboard used throughout: https://github.com/SikamikanikoBG/homelab-monitor TL;DR One 24GB RTX 3090, two GPU services: WhisperX large-v3 (STT, 7.7GB peak) and a Devstral Small 24B email-triage LLM (Q4_K_M, ~18.3GB). 18.3 + 7.7 = 26GB → CUDA OOM whenever they overlapped. The LLM was loaded with a 40k context window but the triage job never needed more than ~5–8k tokens. Capped num_ctx to 8192 → KV cache drops from ~6.1GB to ~1.25GB → model footprint ~18.3GB → ~14.2GB . 14.2 + 7.7 = 21.9GB → both resident, zero OOM, no quality loss. The setup Host : openSUSE, Xeon (56 threads), 125GB RAM, 1x RTX 3090 (24GB) GPU svc : WhisperX large-v3 (speech-to-text) GPU svc : Ollama -> devstral-small-2 (24B, Q4_K_M) for background email triage Both services run all the time. The OOM only happened when I dictated to my assistant (WhisperX) while the triage loop was active. Step 1 — Make the contention measurable nvidia-smi shows instantaneous VRAM. It can't show you which service spiked or when two of them overlapped — and an intermittent OOM is a timing problem. You need per-service VRAM history. I use my own dashboard (homelab-monitor) for this. The relevant view is "AI Models", which attributes VRAM per model server and per loaded model, over a time range, with OOM markers and a capacity ceiling line. What the history showed at the overlap window: Service Peak VRAM Devstral 24B (triage) ~18.3 GB WhisperX large-v3 7.7 GB Total ~26 GB on a 24 GB card If you want to reproduce the measurement, the dashboard runs as a single container: git clone https://github.com/SikamikanikoBG/homelab-monitor cd homelab-monitor docker compose up -d --build # open http://<host>:9800 -> AI Models / GPU views (NVIDI

A homelab box that never sleeps runs 8,760 hours a year. So the spec that decides what it costs you is not the one on the box. It is the one nobody prints: how many watts it pulls sitting at the login prompt doing nothing. I kept hitting this while shopping for a Proxmox node, so I put the measured numbers in one place. More on that at the end. First, why the spec sheet lies to you. TDP is a thermal budget, not a power reading TDP is the heat the cooler has to handle at full tilt. It is a design target for the heatsink, not a measurement of what the chip draws, and it says almost nothing about idle. Your homelab box spends 95%+ of its life idle, so the number that runs up the meter is idle wall power, and that number is never on the product page. The arithmetic is unforgiving. One watt running continuously is 8.76 kWh a year. So the gap between a 7 W box and a 35 W box is not 28 watts, it is about 245 kWh a year, every year, for as long as the box is on. Plug in your own rate to get the dollars; the point is the gap compounds. Where TDP actively misleads you A few measured results from the dataset I'll link below, all from third-party wall-meter readings, not vendor claims: The new N100 wave is genuinely low. A Minisforum UN100C measures 5 to 7 W at idle. Beelink, GMKtec and Trigkey N100 boxes land in the 6 to 10 W range. For a Pi-hole, a few containers and some light VMs, this tier is hard to beat on running cost. AMD mini PCs idle far higher than their marketing suggests. A Minisforum UM790 Pro measures 25 to 45 W at idle. A Beelink SER6 Pro lands at 20 to 35 W. These are fast little machines, but if you picked one expecting "small box, small draw," the meter disagrees, and over a year that delta is real money. Newer and higher-TDP is not lower-idle. A Dell OptiPlex 7060 Micro idles just over 18 W on its 65 W-TDP desktop chip. The older 7070 with a six-core part sits around 13 W, and the low-power "T" SKUs lower still. The CPU's TDP class predicted idle better tha

You buy a NAS for silent, always-on storage. It sits in a corner, humming quietly, doing its thing. Then you install Docker. Then Syncthing. Then a few containers. Suddenly the HDDs never stop. Seeking, spinning, clicking. Not occasionally - constantly. At 2am you can hear it from the next room. This is what happened, why it happens, and how to make it stop. What's Actually Causing the Noise Mechanical HDDs make noise when the read/write head moves. The more random the I/O - small reads and writes scattered across the disk - the more seeking, the more noise. Sequential writes to a single file are quiet. Random I/O across thousands of small files is loud. Docker and Syncthing are both pathological for HDDs in different ways. Docker overlay2 Docker's default storage driver is overlay2. Every container runs on top of layered filesystems - the image layers are stacked, and a thin writable layer sits on top for each running container. Every file operation inside a container that touches a file from a lower layer triggers a copy-on-write : the entire file gets copied up to the writable layer before the write happens. On an SSD this is fast and silent. On spinning HDDs with mechanical heads, every copy-on-write is a seek, a read, and a write - often scattered across the disk. And it's not just copy-on-write. Docker's overlay2 metadata lives in small files across a deep directory tree. Container startup reads dozens of these. Log rotation writes to them. Health checks touch them. Any container doing anything at all generates constant scattered I/O. Syncthing's 60-Second Heartbeat Syncthing uses BoltDB as its internal database - it stores file indexes, sync state, and peer information there. BoltDB flushes to disk every 60 seconds regardless of whether anything changed. On a busy sync folder with thousands of files, this flush isn't a single sequential write. It rewrites pages across the B-tree structure, which means multiple seeks across the database file. Quiet when it's a