标签：#localllm

Scheduled LLM curation is the least glamorous agent workload you run. A cron job wakes up at 3am, reads a pile of memory, asks a model to dedupe it, summarize it, re-rank it, and writes the result back. Nobody is watching. There's no chat window, no streaming tokens, no human to click a button. It just has to work, quietly, every night. That "nobody is watching" part is exactly what makes the cloud-versus-local decision harder than it looks. When you have a human in the loop, a failed API call throws an error you can see and retry. In a headless cron context, the same failure turns into a job that hangs on an approval prompt no one will ever answer, or a pod that curated three months of context into an emptyDir that vanished on restart. I've run curation both ways: nightly jobs hitting a hosted API, and the same logic pointed at a local model on my Kubernetes cluster. Both work. They fail differently, cost differently, and demand different things from you operationally. Here's the actual tradeoff, not the marketing version. The decision point You reach this fork once your agent memory stops being a toy. Early on, you curate by hand or with a cheap synchronous call inside your agent loop. Then the memory grows, the curation gets expensive, and you pull it out into a scheduled job so it runs off the critical path. Now you're paying an API on a timer, and two things start to bug you. First, the data. Curation reads your entire memory store to make decisions. If that memory contains anything you'd rather not stream to a third party (internal notes, customer context, infrastructure details), every scheduled run ships it over the wire. I wrote about the general version of this problem in privacy-routed LLM inference , and scheduled curation is the workload where it bites hardest, because it touches everything, repeatedly, forever. Second, the cost shape. A curation pass over a large vector store is a lot of tokens for a job that produces no user-facing latency benefit. Yo

Intro Day 20! I lined up 10 AIs that turn a single photo into a few seconds of video. Half ran locally on my DGX Spark, half in the cloud 🐱 What I used: DGX Spark (LTX-2.3 / Wan 2.2) / 8 cloud models via fal.ai / ComfyUI / ffmpeg The setup Item Value Input One identical photo (my cat on a desk) Length 6 seconds Settings Identical The only variable The prompt Easy prompt The cat looks at the camera and meows once. It opens its mouth, meows, then closes it. Its tail flicks and its ears twitch. Hard prompt The cat stands upright on its hind legs in a kitchen, wearing a small apron, holding a knife in its front paws and chopping vegetables on a cutting board. Steam rises from a pot behind it. Please, just watch it Some of the cats came out with very long legs. Anyway. First half is the easy prompt, second half the hard one. On the easy prompt, local and cloud were a fair match . On the hard one... cloud, I think...! Three rankings below. Ranking 1: Time Time per 6-second clip on the hard prompt. Rank Model Where Time 🥇 LTX-2.3 Cloud 41s 🥈 Wan 2.7 Cloud 92s 🥉 Happy Horse 1.1 Cloud 97s 4 Veo 3.1 Cloud 128s 5 Kling 3 Pro Cloud 205s 6 Seedance 2.0 Cloud 210s 7 LTX-2.3 Local 315s 8 Wan 2.2 Local 651s 9 daVinci-MagiHuman Cloud 710s 10 HunyuanVideo 1.5 Cloud 796s A 19x spread. Look at 1st and 7th. Same model, LTX-2.3 , nearly the same resolution. The only difference is where it ran — 7.6x . Local setup DGX Spark (GB10, 128GB unified memory, ~273GB/s). ComfyUI headless, workflows over its API. LTX-2.3 is distilled fp8 at 8 steps. At 1088×1920 peak memory hit 77.8GB, about 60% of 128GB. That was the ceiling. Dropping to 512×768 finishes in 70s, but with one-fifth the pixels. Wan 2.2 is I2V-A14B fp8, 20 steps, 480×640. Higher resolution does not finish in reasonable time. Ranking 2: Cost Rank Model Per 6 seconds 🥇 Local Electricity only 🥈 LTX-2.3 (cloud) $0.36 🥉 Wan 2.7 $0.90 4 Kling 3 Pro $1.01 5 Happy Horse 1.1 $1.08 6 Veo 3.1 $2.40 7 Seedance 2.0 $4.09 — HunyuanVideo / MagiHum

I spent three iterations on an auto-fix pipeline that still doesn't work reliably. Here's what I learned. Loop 1 Wrote a background script. Pull tickets from Azure DevOps, run them through a local model, hand to a coding agent, push the result. Poll → triage → fix → push. Worked 40% of the time on trivial tickets. Anything that crossed file boundaries or needed real context — stalled or hallucinated. I shipped it anyway. That was naive. Loop 2 Made it smarter. Pre-selected relevant files. Broke big tickets into subtasks. Turned complex edits into atomic steps with verification between each. Got it to 55% or so. But every fix created two new edge cases. The complexity was compounding faster than the reliability. Loop 3 Went all in. Embeddings for dedup. Multi-repo routing. Auto-revert. A learning loop that fed failures back into future runs. The model server started dying. 890 memory errors in a day. Root cause: two independent consumers hitting the same local model server, each with its own retry loop. When memory filled up, retries amplified instead of staggering. The system was making itself worse. Fixes were simple in hindsight — stop retrying OOM, serialize access, use the local binary not npx. But the pattern kept repeating: add more to fix the last thing, break something else. Where I'm At The pipeline still only works on easy tickets. Hard ones need a human. After three rounds, the main thing I learned is that local models hit a wall before your ambition does — not in quality, in working memory. And adding features doesn't fix reliability gaps. It just moves them around. The 507 retry spiral taught me more than any successful deploy this year. Because it was entirely my fault. Not the model's, not the framework's. I built concurrent consumers with independent retry loops and expected them to coordinate. They didn't. What's Next I'll do a fourth loop. Smaller. A dedicated fast model for cheap work, the big model only for editing. One consumer at a time. Might

This is the technical, reproducible version of a fix I shipped on my own homelab. If you want the narrative version, that's on Medium. This one is the recipe: the measurements, the math, the Modelfile, and the exact prompt I gave Claude Code to generate it. Copy-paste friendly. Repo for the dashboard used throughout: https://github.com/SikamikanikoBG/homelab-monitor TL;DR One 24GB RTX 3090, two GPU services: WhisperX large-v3 (STT, 7.7GB peak) and a Devstral Small 24B email-triage LLM (Q4_K_M, ~18.3GB). 18.3 + 7.7 = 26GB → CUDA OOM whenever they overlapped. The LLM was loaded with a 40k context window but the triage job never needed more than ~5–8k tokens. Capped num_ctx to 8192 → KV cache drops from ~6.1GB to ~1.25GB → model footprint ~18.3GB → ~14.2GB . 14.2 + 7.7 = 21.9GB → both resident, zero OOM, no quality loss. The setup Host : openSUSE, Xeon (56 threads), 125GB RAM, 1x RTX 3090 (24GB) GPU svc : WhisperX large-v3 (speech-to-text) GPU svc : Ollama -> devstral-small-2 (24B, Q4_K_M) for background email triage Both services run all the time. The OOM only happened when I dictated to my assistant (WhisperX) while the triage loop was active. Step 1 — Make the contention measurable nvidia-smi shows instantaneous VRAM. It can't show you which service spiked or when two of them overlapped — and an intermittent OOM is a timing problem. You need per-service VRAM history. I use my own dashboard (homelab-monitor) for this. The relevant view is "AI Models", which attributes VRAM per model server and per loaded model, over a time range, with OOM markers and a capacity ceiling line. What the history showed at the overlap window: Service Peak VRAM Devstral 24B (triage) ~18.3 GB WhisperX large-v3 7.7 GB Total ~26 GB on a 24 GB card If you want to reproduce the measurement, the dashboard runs as a single container: git clone https://github.com/SikamikanikoBG/homelab-monitor cd homelab-monitor docker compose up -d --build # open http://<host>:9800 -> AI Models / GPU views (NVIDI