标签：#local

Researchers from UC Berkeley and MIT have developed FreeToken, an open-source inference engine that enhances the utility of Mixture-of-Experts models on consumer hardware. By implementing a dynamic scheduling policy and optimising weight management, FreeToken improves decoding speeds and execution efficiency in edge AI applications, fostering self-hosted reasoning systems. By Olimpiu Pop

Meta AI Research has introduced Muse Glimmer, a 30-billion-parameter open-weight model under the Apache 2.0 license, designed for local workflows. It enables autonomous agents and complex task execution on consumer GPUs without relying on cloud APIs. The model employs a multi-stage training approach for efficient performance and supports multimodal inputs, enhancing coding and automation tasks. By Olimpiu Pop

Scheduled LLM curation is the least glamorous agent workload you run. A cron job wakes up at 3am, reads a pile of memory, asks a model to dedupe it, summarize it, re-rank it, and writes the result back. Nobody is watching. There's no chat window, no streaming tokens, no human to click a button. It just has to work, quietly, every night. That "nobody is watching" part is exactly what makes the cloud-versus-local decision harder than it looks. When you have a human in the loop, a failed API call throws an error you can see and retry. In a headless cron context, the same failure turns into a job that hangs on an approval prompt no one will ever answer, or a pod that curated three months of context into an emptyDir that vanished on restart. I've run curation both ways: nightly jobs hitting a hosted API, and the same logic pointed at a local model on my Kubernetes cluster. Both work. They fail differently, cost differently, and demand different things from you operationally. Here's the actual tradeoff, not the marketing version. The decision point You reach this fork once your agent memory stops being a toy. Early on, you curate by hand or with a cheap synchronous call inside your agent loop. Then the memory grows, the curation gets expensive, and you pull it out into a scheduled job so it runs off the critical path. Now you're paying an API on a timer, and two things start to bug you. First, the data. Curation reads your entire memory store to make decisions. If that memory contains anything you'd rather not stream to a third party (internal notes, customer context, infrastructure details), every scheduled run ships it over the wire. I wrote about the general version of this problem in privacy-routed LLM inference , and scheduled curation is the workload where it bites hardest, because it touches everything, repeatedly, forever. Second, the cost shape. A curation pass over a large vector store is a lot of tokens for a job that produces no user-facing latency benefit. Yo

Intro Day 20! I lined up 10 AIs that turn a single photo into a few seconds of video. Half ran locally on my DGX Spark, half in the cloud 🐱 What I used: DGX Spark (LTX-2.3 / Wan 2.2) / 8 cloud models via fal.ai / ComfyUI / ffmpeg The setup Item Value Input One identical photo (my cat on a desk) Length 6 seconds Settings Identical The only variable The prompt Easy prompt The cat looks at the camera and meows once. It opens its mouth, meows, then closes it. Its tail flicks and its ears twitch. Hard prompt The cat stands upright on its hind legs in a kitchen, wearing a small apron, holding a knife in its front paws and chopping vegetables on a cutting board. Steam rises from a pot behind it. Please, just watch it Some of the cats came out with very long legs. Anyway. First half is the easy prompt, second half the hard one. On the easy prompt, local and cloud were a fair match . On the hard one... cloud, I think...! Three rankings below. Ranking 1: Time Time per 6-second clip on the hard prompt. Rank Model Where Time 🥇 LTX-2.3 Cloud 41s 🥈 Wan 2.7 Cloud 92s 🥉 Happy Horse 1.1 Cloud 97s 4 Veo 3.1 Cloud 128s 5 Kling 3 Pro Cloud 205s 6 Seedance 2.0 Cloud 210s 7 LTX-2.3 Local 315s 8 Wan 2.2 Local 651s 9 daVinci-MagiHuman Cloud 710s 10 HunyuanVideo 1.5 Cloud 796s A 19x spread. Look at 1st and 7th. Same model, LTX-2.3 , nearly the same resolution. The only difference is where it ran — 7.6x . Local setup DGX Spark (GB10, 128GB unified memory, ~273GB/s). ComfyUI headless, workflows over its API. LTX-2.3 is distilled fp8 at 8 steps. At 1088×1920 peak memory hit 77.8GB, about 60% of 128GB. That was the ceiling. Dropping to 512×768 finishes in 70s, but with one-fifth the pixels. Wan 2.2 is I2V-A14B fp8, 20 steps, 480×640. Higher resolution does not finish in reasonable time. Ranking 2: Cost Rank Model Per 6 seconds 🥇 Local Electricity only 🥈 LTX-2.3 (cloud) $0.36 🥉 Wan 2.7 $0.90 4 Kling 3 Pro $1.01 5 Happy Horse 1.1 $1.08 6 Veo 3.1 $2.40 7 Seedance 2.0 $4.09 — HunyuanVideo / MagiHum

I Built an Agent Evaluation Harness for Local AI — Here's What Most People Get Wrong DOYR | Not financial/legal/tax advice. For educational purposes only. Three months ago, I started building AI agents for my trading business. First agent: Fetches Nifty option chain data. Second agent: Analyzes PCR, OI, max pain. Third agent: Predicts direction using XGBoost. Fourth agent: Sends Telegram alerts. I had 4 agents doing 5 jobs. And I had no idea if they were any good . Sure, my trading results were +₹96,000 over 6 months. But was that because my agents were smart, or because I was overriding their bad decisions? I couldn't answer that question. So I built something to find out. An Agent Evaluation Harness. What Is an Agent Evaluation Harness? An Agent Evaluation Harness is a systematic framework for testing AI agents. It answers one question: "How good is this agent, actually?" Most people skip evaluation. They build an agent, test it once or twice manually, and call it "done." Then they wonder why it fails in production. An evaluation harness forces you to: Define success metrics — what does "good" mean? Create test suites — what scenarios will you test? Run evaluations — how does the agent perform across all scenarios? Measure regressions — did a change make the agent worse? Track improvements — is version 2 better than version 1? This is not optional. This is engineering 101 . Why Most Agent Evaluations Are Wrong I reviewed 50+ "agent evaluation" frameworks online. Here's what I found: Mistake 1: Single-Task Testing What they do: Test the agent on one task. "Can it book a flight?" → Yes/No. What's wrong: Real agents face thousands of variations of the same task. "Book a flight from Delhi to Mumbai on Friday" vs "Book a flight from Delhi to Mumbai next Friday" vs "Book a flight from Delhi to Mumbai on August 15th." A good harness tests variations , not just one example. Mistake 2: No Edge Cases What they do: Test happy paths only. "Book a flight when everything works.

Part 1 established the hardware, the runner, and the primary model. This entry covers what governs inference on that hardware — the two phases of inference, the cost of long context, and the cost of loading a model from disk — and compares the local machines against a free-tier cloud model. The two phases of inference Inference has two phases. Prefill processes the input prompt before any output appears; it is compute-bound and wants a GPU. Generation produces output tokens one at a time and is bound by memory bandwidth. Casual use is almost all generation and hides the difference; the cost of prefill surfaces only when prompts grow large. The machines, and how they were measured Machine CPU / RAM GPU (VRAM) Storage (read) Prefill (tok/s) Gen (tok/s) Load (18 GB) Primary desktop 5950X / ~80 GB DDR4 RX 6900XT (16 GB) NVMe (~2.1 GB/s) 360 18.3 8.4s Secondary box 5600G / 32 GB DDR4 GTX 1060 (6 GB) SATA SSD (~0.35 GB/s) 253 17.1 50.6s Laptop 8840U / 32 GB DDR5 Radeon 780M (none) NVMe (~2.4 GB/s) 20 10.0 7.5s All inference figures come from a controlled run: the same model (Gemma 4 26B, 18 GB) on each machine, a unique random prefix per prompt to defeat caching, a fixed 8,192-token context, warm, on an identical ~6,855-token prompt (generation timed over a 200-token output). Two things stand out. Prefill varies about eighteen-fold across the machines (360 to 20 tok/s) while generation varies less than twofold (18.3 to 10.0), and prefill is what dominates large-prompt workloads — so a machine can look fine on generation yet be useless in practice. Model-load time, separately, is set by storage rather than compute: the secondary box's budget SATA SSDs load the 18 GB model in 50 seconds against eight on NVMe, which turns a cold request into a minute-long stall. Secondary box Request time Warm (model resident) ~4s Cold (model reload) ~54s If the model is allowed to unload between calls, every call silently pays that reload — a real source of intermittent timeouts. The fix is

This is the first in a series of build-log posts documenting a local LLM project, in which models are run on owned consumer hardware rather than through a cloud API. The present entry covers the hardware, the software stack, and the benchmarks by which a primary model was selected. The hardware Two machines are used, both consumer-grade. All benchmarks reported below were obtained on the primary desktop. Machine CPU RAM GPU Primary desktop Ryzen 5950X ~80 GB DDR4 AMD RX 6900XT (16 GB) Secondary box Ryzen 5600G 32 GB NVIDIA GTX 1060 (6 GB) The software stack Ollama serves as the model runner across two GPU vendors: ROCm 5.7 for the AMD card on the primary desktop, and CUDA for the NVIDIA card on the secondary box. The primary model is Gemma 4 26B, a mixture-of-experts model with roughly 3.8B active parameters, quantized to Q4_K_M and occupying approximately 18 GB on disk. On the RX 6900XT it is run with an automatic GPU/CPU layer split, as the Q4 weights together with the KV cache exceed the 16 GB of available VRAM. Several Ollama settings were enabled to recover headroom: flash attention, and an 8-bit ( q8_0 ) KV cache, the latter approximately halving the cache footprint. A free cloud tier is retained for occasional heavier tasks, though the objective is to run as much as possible locally. Selecting a model: benchmarks Before a primary model was chosen, the installed models were benchmarked. Two properties were of interest: throughput and output quality. Throughput was measured on the primary desktop with a 500-word essay prompt ( ollama run <model> --verbose ): Model Tokens/sec Duration Tokens out gemma4:26b 18.86 50.11s 945 gemma4-26b (64K ctx) 17.96 51.99s 934 mistral:7b-instruct 34.81 10.17s 354 llama3.2 57.11 3.99s 228 The smaller models are substantially faster; their token counts, however, are lower, and in practice their responses were correspondingly shallower. Quality was assessed with a five-task suite spanning logic, coding, summarization, creative writ

JDK 24 removed the monitor-related carrier-thread pinning that stalled Netflix and similar teams on Java 21. What has replaced it on JDK 25 LTS is downstream-resource saturation: The bottleneck moved and now demands explicit bounding in application code. This article maps the failure modes that surface after virtual-thread adoption and gives a practical sequence backed by a public benchmark. By Sandeep Bharadwaj

Most AI “apps” these days aren’t really apps. They’re wrappers. A thin UI. A subscription. A cloud call. A monthly fee. And your data quietly piped off to a server farm you’ll never see. I didn’t want to build another one of those. I didn’t want to use another one of those. So, I built something different. I built Bob’s Bar — the first Collaborative Multi-Persona Sandbox (CMPS). Let's be honest, every AI chat feels the same. You ask a question, it answers, repeat. Useful? Yes. Alive? No. I kept thinking: “Why does every AI tool assume I want a one-on-one conversation? Why can’t multiple AIs talk to each other while I watch?” That question became the seed. I wanted a space where multiple personas could exist together, talk to me and each other, and run entirely on my own hardware. I’m an indie dev. I don’t have a server farm, and I don’t want to pay AWS just to host my own brainstorming sessions. So the architecture is 100% local-first: -Engine: Ollama (raw LLM inference) -UI: Python + Gradio (Blocks API for stateful UI) -Distribution: PyInstaller → standalone .exe -Licensing: Lemon Squeezy API (so I could actually sell the damn thing) No cloud. No subscriptions. No data scraping. Just your machine doing the work. But getting four local models to talk to each other without hallucinating is absolute chaos. At first, they spoke twice in a row. They impersonated each other. They put words in the user’s mouth. They derailed into movie scripts. Vane pitched a heist film, and Bob started giving stage directions like “I polish a glass and chuckle.” I had to build what I now call The Bouncer — a regex-powered clean_reply() function that physically chops off AI responses the moment they write a script, impersonate another persona, use bullet points, or hijack the conversation. I also added a seen_in_banter set to prevent any persona from dominating the room. Once the routing was locked down, the magic happened. Because the AIs share a context window, emergent behaviour happen

Both vulnerabilities allow untrusted users to gain root privileges.

I spent three iterations on an auto-fix pipeline that still doesn't work reliably. Here's what I learned. Loop 1 Wrote a background script. Pull tickets from Azure DevOps, run them through a local model, hand to a coding agent, push the result. Poll → triage → fix → push. Worked 40% of the time on trivial tickets. Anything that crossed file boundaries or needed real context — stalled or hallucinated. I shipped it anyway. That was naive. Loop 2 Made it smarter. Pre-selected relevant files. Broke big tickets into subtasks. Turned complex edits into atomic steps with verification between each. Got it to 55% or so. But every fix created two new edge cases. The complexity was compounding faster than the reliability. Loop 3 Went all in. Embeddings for dedup. Multi-repo routing. Auto-revert. A learning loop that fed failures back into future runs. The model server started dying. 890 memory errors in a day. Root cause: two independent consumers hitting the same local model server, each with its own retry loop. When memory filled up, retries amplified instead of staggering. The system was making itself worse. Fixes were simple in hindsight — stop retrying OOM, serialize access, use the local binary not npx. But the pattern kept repeating: add more to fix the last thing, break something else. Where I'm At The pipeline still only works on easy tickets. Hard ones need a human. After three rounds, the main thing I learned is that local models hit a wall before your ambition does — not in quality, in working memory. And adding features doesn't fix reliability gaps. It just moves them around. The 507 retry spiral taught me more than any successful deploy this year. Because it was entirely my fault. Not the model's, not the framework's. I built concurrent consumers with independent retry loops and expected them to coordinate. They didn't. What's Next I'll do a fourth loop. Smaller. A dedicated fast model for cheap work, the big model only for editing. One consumer at a time. Might

Dai metadati in una lingua a 20 localizzazioni senza impazzire tra click e schermate: un flusso pratico per indie e piccoli team. Localizzare un’app non significa solo tradurre le stringhe dell’interfaccia. Una buona parte dell’acquisizione organica passa dai metadati su App Store Connect : titolo, sottotitolo, descrizione e keyword. Il problema è che, quando provi a farlo “a mano” dal pannello web, diventa subito un lavoro di pura resistenza: apri la scheda, cambi lingua, compili i campi, salvi, ripeti. Ora moltiplica per 10–20 lingue. Per molti indie (e in generale per chi ha poco tempo e zero voglia di click ripetitivi) il punto di svolta è usare ASC CLI per rendere questa attività automatizzabile, ripetibile e verificabile . Perché la localizzazione dei metadati è un caso d’uso perfetto per una CLI Dal punto di vista del flusso di lavoro, i metadati App Store hanno tre caratteristiche che li rendono ideali per l’automazione: Sono campi strutturati (title, subtitle, description, keywords): non stai “inventando” contenuti ogni volta, stai trasformando contenuti. Sono ripetitivi per lingua : la sequenza di operazioni è identica, cambia solo la locale. Sono tanti : più lingue aggiungi, più l’approccio manuale scala male (tempo, errori, incoerenze). Con una CLI, invece, il lavoro si sposta dal “fare cose” al definire un processo : prendi i metadati di partenza, generi le varianti linguistiche, applichi l’update in batch. Cosa conviene localizzare (e cosa no) In genere ha senso includere in un passaggio di localizzazione “massiva”: App name / title (attenzione ai limiti e ai trademark) Subtitle (spesso è la parte più ASO-oriented) Description (qui conta più la leggibilità che la traduzione letterale) Keywords (campo delicato: va adattato, non tradotto alla cieca) Al contrario, è meglio trattare con più cautela: Claim e frasi marketing molto creative : in alcune lingue risultano innaturali se tradotte letteralmente Keyword strategy : la ricerca utenti cambia per mercat

LLM costs scale linearly with usage. A system processing 10,000 requests a day at $0.01 per request costs $100 daily — $365 a year. At enterprise scale, that's over $10,000. Cost optimization isn't about cutting corners. It's about spending tokens where they matter. Every token you waste is a token you could have spent on a better answer. Token budgeting The simplest way to control costs is to set limits. Per session, per task, or per day. Strategy 1: Per-Session Budgets Per-session budgets are straightforward: class SessionBudget : def __init__ ( self , budget_tokens : int = 10000 ): self . budget = budget_tokens self . used = 0 def allocate ( self , tokens : int ) -> bool : if self . used + tokens <= self . budget : self . used += tokens return True return False def remaining ( self ) -> int : return self . budget - self . used Strategy 2: Per-Task Budgets Per-task budgets are more useful. Different tasks need different amounts of context: task_budgets : classify : max_tokens : 100 model : qwen2.5-1.5b summarize : max_tokens : 500 model : qwen2.5-7b code_review : max_tokens : 2000 model : qwen2.5-coder-7b reason : max_tokens : 4000 model : qwen2.5-32b Strategy 3: Adaptive Budgets Adaptive budgets adjust based on what actually happens. If classification tasks consistently use 80 tokens, stop allocating 100: class AdaptiveBudget : def __init__ ( self ): self . task_history = {} def allocate ( self , task_type : str ) -> int : if task_type in self . task_history : return int ( self . task_history [ task_type ] * 1.5 ) return 1000 def record ( self , task_type : str , tokens_used : int ): if task_type not in self . task_history : self . task_history [ task_type ] = tokens_used else : self . task_history [ task_type ] = ( 0.9 * self . task_history [ task_type ] + 0.1 * tokens_used ) The exponential moving average (0.9 weight) means recent usage matters more than history. Adjust the weight based on how volatile your workloads are. API vs local inference Local inference

Running a 70B parameter model to summarize a 200-word email is wasteful. Running a 3B model to review production code is reckless. Most systems live somewhere in between — and that's where model routing comes in. It matches task complexity to model capability. The tradeoffs are real, but the savings are too. The routing problem People usually start with one model and stick with it. That works until you notice the cost, or the latency, or both. The alternative is building a router — something that decides which model handles which request. Four strategies work in practice: Capability-based — route by what the model can do Cost-aware — route by what you're willing to spend Latency-aware — route by how fast you need it Hybrid — combine them Each optimizes something different. Picking one is usually a decision about what hurts most. Capability-based routing The simplest approach. Classify the task, send it to the model that handles it. Task Model size Examples Classification, tagging 1-3B Qwen2.5-1.5B, Gemma-2-2B Summarization, extraction 3-7B Qwen2.5-7B, Llama-3.1-8B Code generation 7-14B Qwen2.5-Coder-7B, DeepSeek-Coder-V2 Complex reasoning 14-32B Qwen2.5-32B, Llama-3.1-70B Creative writing, analysis 32B+ Qwen2.5-72B, Claude, GPT-4 If the task doesn't need the bigger model, don't use it. A 1.5B model handles sentiment classification fine. It just won't write a coherent essay. Implementation is straightforward: ROUTING_RULES = { " classify " : { " model " : " qwen2.5-1.5b " , " max_tokens " : 100 }, " summarize " : { " model " : " qwen2.5-7b " , " max_tokens " : 500 }, " code_review " : { " model " : " qwen2.5-coder-7b " , " max_tokens " : 2000 }, " reason " : { " model " : " qwen2.5-32b " , " max_tokens " : 4000 }, " creative " : { " model " : " claude-sonnet-4 " , " max_tokens " : 8000 }, } def route_request ( task_type : str ) -> dict : return ROUTING_RULES . get ( task_type , ROUTING_RULES [ " reason " ]) The catch is classification itself. If you get the task type

Here's a thing that catches almost everyone the first week they run a model locally. You paste a 600-line file into your shiny new local assistant, ask it to find the bug, and it confidently rewrites a function that isn't even in the part it read. No error. No warning. It just... silently dropped most of your file on the floor before the model ever saw it. That's not the model being dumb. That's Ollama doing exactly what it was told. By default it gives every model a context window of 2048 tokens and quietly truncates anything past that. It's one of a handful of small surprises that separate "I installed Ollama" from "I actually understand what's running on my machine." Let's go through the ones that matter: how the thing works under the hood, what hardware you really need, the gotchas, and the honest answer to "should I even bother instead of just calling an API?" What Ollama actually is Ollama gets described as "Docker for LLMs," and that's a decent first approximation. You pull a model, you run it, there's a registry. But it hides what's doing the heavy lifting. Underneath, Ollama is a friendly wrapper around llama.cpp , the C/C++ inference engine that made running these models on consumer hardware practical in the first place. When you type ollama run , you're really booting a llama.cpp runtime with a sane default config and a tidy HTTP server bolted on. The models it runs are in a format called GGUF (GPT-Generated Unified Format). A GGUF file isn't just weights. It's a self-contained package that bundles the tensors, the tokenizer config, the architecture details, and hyperparameters like the trained context length, all in one file. That's why ollama pull llama3.1 gives you something that just works: everything the runtime needs to reconstruct the model is in the box. Ollama itself is young. The project shipped its first release in early July 2023 , and it rode the wave of open-weight models (Llama 2 landed that same month) that suddenly made "run a real LLM on

This post is part of the akm-knowledge series. Part ten introduced the improve pipeline — what each phase does and how to schedule it. This post goes deeper on what continuous operation looks like in practice: the hardware numbers, the reliability bugs we hit at 48 runs per day, and the observability layer we built to keep watch. Most people think of AI agent memory as something that happens during a session. You talk to your agent, it learns things, maybe you save a few notes, the session ends. The next session starts cold. akm improve is built around a different model: a continuous background process that runs on your own hardware, against local models, and quietly curates your agent's knowledge base while you work on other things. No cloud API required. No per-token billing for the maintenance pass. A GPU you already own, a model you already have downloaded, running on a schedule. This post covers what 24 hours of autonomous operation actually looks like, how consumer-grade GPUs handle the load, the reliability work that makes continuous operation viable, and the observability layer that lets you know it's working without watching logs. What akm improve Does in 24 Hours akm improve is a multi-phase pipeline. The core pass — consolidation — loads your memory pool, groups related memories into chunks, sends each chunk to a local LLM for a consolidation plan (merge similar memories, promote high-signal ones to your stash, delete redundant ones, surface contradictions), and then executes those plans. After consolidation, memory inference runs a lightweight factual extraction pass, and graph extraction updates the entity-relation index. The pipeline is scheduled to run automatically. Here is what one 24-hour window produced: Metric Value Runs completed 48 / 48 — zero failures Memories processed 14,189 Promoted to stash 1,361 Merged (deduplication) 49 (64 secondaries absorbed) Contradictions surfaced 211 Deleted (redundant) 31 Memory inference yield 69.3% — 115 new ato

This is part thirteen in a series about managing the growing pile of skills, scripts, and context that AI coding agents depend on. Part ten covered the full improve pipeline — all five phases and how they connect. Part fourteen covers what 48 runs per day looks like in practice, including hardware benchmarks and the reliability bugs that surface at that frequency. The reflect pass inside akm improve has three execution modes. Most installs are still running the slowest one. Agent mode — the original — spawns an opencode or claude subprocess for each reflect call. The subprocess starts cold, acquires a session, assembles context, makes its LLM call, and exits. That cold-start overhead is real: each call takes approximately 30 seconds on a quiet machine. Run akm improve against a 69-ref stash and the reflect phase alone costs about 35 minutes. SDK mode eliminated the subprocess. The reflect call runs in-process, cutting per-call latency to 10–15 seconds. A 69-ref run drops to 12–17 minutes — better, but still bounded by round-trip overhead that the reflect task does not actually need. LLM mode removes the round trip entirely. The context for reflect is statically pre-assembled — no live tool calls, no file reads, no external context needed. A direct HTTP call to the LLM endpoint is sufficient, and it costs 6–10 seconds per call. A 69-ref run completes in 8–10 minutes. Mode Per-call latency 69-ref run agent (CLI subprocess) ~30s ~35 min sdk (in-process) ~10–15s ~12–17 min llm (direct HTTP) ~6–10s ~8–10 min The 3–4× end-to-end improvement is from eliminating overhead that was never necessary for what reflect does. Why Reflect Does Not Need an Agent The reflect pass takes a stash asset, examines its current content, and proposes a refined version. The inputs are fixed before the pass starts: the asset text, its metadata, and the improvement prompt. Nothing changes mid-call. No files need to be opened. No search queries need to fire. No external context needs to be pulled

This is the technical, reproducible version of a fix I shipped on my own homelab. If you want the narrative version, that's on Medium. This one is the recipe: the measurements, the math, the Modelfile, and the exact prompt I gave Claude Code to generate it. Copy-paste friendly. Repo for the dashboard used throughout: https://github.com/SikamikanikoBG/homelab-monitor TL;DR One 24GB RTX 3090, two GPU services: WhisperX large-v3 (STT, 7.7GB peak) and a Devstral Small 24B email-triage LLM (Q4_K_M, ~18.3GB). 18.3 + 7.7 = 26GB → CUDA OOM whenever they overlapped. The LLM was loaded with a 40k context window but the triage job never needed more than ~5–8k tokens. Capped num_ctx to 8192 → KV cache drops from ~6.1GB to ~1.25GB → model footprint ~18.3GB → ~14.2GB . 14.2 + 7.7 = 21.9GB → both resident, zero OOM, no quality loss. The setup Host : openSUSE, Xeon (56 threads), 125GB RAM, 1x RTX 3090 (24GB) GPU svc : WhisperX large-v3 (speech-to-text) GPU svc : Ollama -> devstral-small-2 (24B, Q4_K_M) for background email triage Both services run all the time. The OOM only happened when I dictated to my assistant (WhisperX) while the triage loop was active. Step 1 — Make the contention measurable nvidia-smi shows instantaneous VRAM. It can't show you which service spiked or when two of them overlapped — and an intermittent OOM is a timing problem. You need per-service VRAM history. I use my own dashboard (homelab-monitor) for this. The relevant view is "AI Models", which attributes VRAM per model server and per loaded model, over a time range, with OOM markers and a capacity ceiling line. What the history showed at the overlap window: Service Peak VRAM Devstral 24B (triage) ~18.3 GB WhisperX large-v3 7.7 GB Total ~26 GB on a 24 GB card If you want to reproduce the measurement, the dashboard runs as a single container: git clone https://github.com/SikamikanikoBG/homelab-monitor cd homelab-monitor docker compose up -d --build # open http://<host>:9800 -> AI Models / GPU views (NVIDI

By Takeshi Yokoyama — Onecarat Labs Hi. I'm Yokoyama, and I build a local-first AI text editor as a side project, along with a few other experimental tools. Working on them, I keep running into the same question about where the web is going. This post is one observation, plus a small experiment I built to test it — including a Chrome extension you can actually try. The short version: I think websites will increasingly be read through AI agents, reshaped per reader, on the fly. And once that happens, there's a clear gap between sites that are easy for an AI to read and sites that aren't. What's starting to happen Until now, people read websites as websites. You open the top page, follow the menu, read the body, click a button — tracing the path the maker designed. As local AI and AI agents become normal, that breaks. People stop opening the page directly. They tell an AI what they want — "Can I try this quickly?" , "I just want to check it's safe" , "Just the gist" — and the AI reads the web and reshapes it into the form that reader wants. What the reader receives is no longer the layout the maker built. This isn't speculation. The idea that AI generates the interface for the reader already has a name — Generative UI — and it's one of the hottest areas in frontend right now, with Google, Vercel and others building toward it. But notice who's holding the pen in almost every version of that story: the site , or an AI embedded in an app — something under the maker's control. What I'm looking at is one step past that: a local AI, in the reader's own hands, reshaping any site into that person's preferred form — with no involvement from the maker at all. The initiative moves from the maker to the reader. The part that nags at me as a builder I build software too. So this shift nags at me. A site carries its maker's intent and rights. The order things appear in, what gets emphasized, the tone. Design, copy, flow — all of it is deliberate. Having an AI quietly reorder, rewri