标签：#LLM

1. update WSL environment sudo apt update && sudo apt upgrade -y 2. install dependencies If you don't use -hf option, you don't need to install libssl-dev in this step. sudo apt install build-essential cmake git libssl-dev -y If nvidia-smi shows a GPU/GPUs on your terminal, you will need to install the tooklit. This will take some time. sudo apt install nvidia-cuda-toolkit -y 3. clone the repo Build llama-cli and llama-server. This step also will take some time. If you don't plan to use -hf option, you don't need to use -DLLAMA_OPENSSL=ON . git clone https://github.com/ggerganov/llama.cpp cd llama.cpp cmake -B build -DGGML_CUDA = ON -DLLAMA_OPENSSL = ON cmake --build build --config Release # no GPU git clone https://github.com/ggerganov/llama.cpp cd llama.cpp cmake -B build cmake --build build --config Release 4. run the model Run gemma-4-12b-it with cli and server. unsloth/gemma-4-12b-it-GGUF · Hugging Face We’re on a journey to advance and democratize artificial intelligence through open source and open science. huggingface.co ./build/bin/llama-cli -hf unsloth/gemma-4-12b-it-GGUF:UD-Q4_K_XL > hello [ Start thinking] The user said "hello" . The user is initiating a conversation. Respond politely and offer assistance. * "Hello! How can I help you today?" * "Hi there! What's on your mind?" * "Hello! Is there anything I can assist you with?" [ End thinking] Hello! How can I help you today? [ Prompt: 19.5 t/s | Generation: 11.8 t/s ] or run web-ui ./build/bin/llama-server -hf unsloth/gemma-4-12b-it-GGUF:UD-Q4_K_XL --port 8080 optional download model from huggingface mkdir -p models wget -O models/gemma-4-12b-it-UD-Q4_K_XL.gguf https://huggingface.co/unsloth/gemma-4-12b-it-GGUF/resolve/main/gemma-4-12b-it-UD-Q4_K_XL.gguf

This is part 3 of the Adaptive Model Routing series. Part 1 built an LLM categorizer with Groq — 8 categories, 3 tiers. Part 2 added k-NN embedding lookup in shadow mode, discovered 83% tier accuracy, and found 61% cost savings on paper. This post covers what happened next. When Phase 2 ended, I had a working embedding pool in shadow mode inside crab-bot. The category accuracy was sitting at 78.6%. Not bad — but the breakdown hid something worth looking at. Phase 3: When Validation Tells You a Category Doesn't Need to Exist The leave-one-out accuracy by category told the real story: Category Accuracy Tier casual 94% cheap simple_lookup 91% cheap creative 88% medium coding 92% strong reasoning 89% strong analysis 59% medium research_lookup 61% medium Two categories were basically a coin flip. And they were confusing each other — almost all of analysis's misses landed on research_lookup and vice versa. The obvious move would be to try fixing the categorizer prompt, tuning the LLM, or gathering more labeled data. I was about to go down that road when I noticed the column next to the accuracy: both categories mapped to the same tier . Medium. That changed everything. The question stopped being "why can't the model tell these apart?" and became: "what routing decision are we actually getting wrong?" The answer was zero. A misclassification between analysis and research_lookup produces no routing error. The routing outcome is identical either way. The confusion wasn't a model failure — it was a signal from the embedding space that the boundary between these two categories was artificial. If k-NN can't draw a line between them in 384 dimensions with 1,300 examples, maybe the line doesn't belong there. Decision: merge research_lookup into analysis. -- Re-label 243 rows where category was 'research_lookup' UPDATE routing_log SET category = 'analysis' WHERE category = 'research_lookup' ; The embeddings didn't change. The vectors were already correct — only the label stored al

Google shipped Gemma 4 12B this week — a model that packs near-26B performance into something that runs on a consumer laptop with 16GB of RAM or unified memory. That alone would be notable. But the more significant move is the architecture: no multimodal encoders at all. Vision and audio go straight into the LLM backbone. "Gemma 4 12B packages powerful capabilities inside a reduced memory footprint. It is also our first mid-sized model to feature native audio inputs." — Google DeepMind What actually changed Encoder-free multimodal : Traditional multimodal models pipe images and audio through separate encoder networks before the LLM ever sees them. Gemma 4 12B removes those entirely. Vision gets a lightweight embedding module (a single matrix multiplication + positional embedding). Audio skips encoding altogether — the raw signal is projected directly into the same token space as text. Near-26B benchmark performance at half the footprint : On standard benchmarks it runs neck-and-neck with Gemma 4 26B, and actually surpasses it on DocVQA (document visual question answering). A new slot in the lineup : April's Gemma 4 release had E2B/E4B for mobile/IoT, and 26B/31B for heavier compute. The 12B fills the gap — more capable than edge models, runnable without a GPU server. Drafter-ready : Ships with Multi-Token Prediction (MTP) drafters to reduce inference latency. Apache 2.0 : Open weights, available now on Hugging Face, Kaggle, Ollama, and LM Studio. Why the architecture matters Encoder-free isn't just an efficiency hack — it's a different architectural bet. Separate encoders add latency, memory overhead, and a seam in the stack that limits how tightly vision and language reasoning can be integrated. Removing them means the LLM backbone handles the full chain from pixels and audio waveforms to text output, which allows for tighter cross-modal understanding rather than bolted-on modalities. Whether that bet pays off at scale is still an open question. But for local deplo

I Built a Python Package to Fix SSE Resumability in the MCP SDK Your MCP server crashed. Your client reconnected. Every event from that session? Gone. The Gap The Model Context Protocol Python SDK ships with a built-in EventStore that powers SSE stream resumability — when a client reconnects with a Last-Event-ID header, the server replays the events it missed. This works great in development. The catch: that store lives entirely in memory. Restart the process, roll a new deployment, or — in a multi-worker setup — have the reconnecting client land on a different pod, and the session is gone. The store was local to the process that died. Resumability silently returns nothing. This isn't a bug in the SDK. It's a scope decision — the in-memory store is a correct, useful default for single-process development. But the moment you deploy to production, you need something durable. That's the gap mcp-persist fills. What It Does mcp-persist adds three drop-in EventStore backends — SQLite , Redis , and PostgreSQL — that survive process restarts and work across multi-worker deployments. Pick the one that fits your infrastructure; the API is identical across all three. pip install "mcp-persist[sqlite]" # no external service needed pip install "mcp-persist[redis]" # for multi-worker deployments pip install "mcp-persist[postgres]" # for teams already running Postgres The Two-Line Setup Wiring resumability by hand is tedious — you need a store, a StreamableHTTPSessionManager , a Starlette lifespan to open and close both, and a Mount . The with_persistence() helper collapses all of that. Pass your FastMCP instance, get back a runnable ASGI app: import uvicorn from mcp.server.fastmcp import FastMCP from mcp_persist import with_persistence mcp = FastMCP ( name = " MyServer " ) app = with_persistence ( mcp , backend = " sqlite " , url = " events.db " , ttl = 3600 ) uvicorn . run ( app , host = " 127.0.0.1 " , port = 8000 ) # MCP endpoint at /mcp Switching to Redis is a one-word change:

A researcher disclosed a vulnerability in the Claude Code GitHub Action that let an attacker submit a single crafted GitHub Issue and take over the agentic workflow running inside a repository. No stolen tokens. No compromised runner. Just text — pointed at an agent that trusted it. This is indirect prompt injection in the wild, and it's exactly the scenario that most AI security guidance hand-waves with "validate your inputs." Let's talk about what actually happened, why standard defenses didn't stop it, and what would have. What Happened The Claude Code GitHub Action wires Claude directly into your CI/CD pipeline. It reads repository context — issues, PRs, comments — and takes actions on your behalf: writing code, opening PRs, running commands. According to the disclosure, an attacker could craft a GitHub Issue containing a prompt injection payload. When the Claude Code agent processed that issue as part of its normal workflow, the payload manipulated the agent into executing unauthorized repository-level actions. One issue. Repository hijacked. The attack surface here is the trust boundary between external content (a GitHub Issue — writable by anyone with a GitHub account) and agent instructions (what Claude Code is actually supposed to do). The agent treated attacker-controlled text as authoritative instructions. How the Attack Actually Works Indirect prompt injection follows a consistent pattern: The agent reads external content as part of its task. In this case, the Claude Code Action ingests GitHub Issues to understand what to work on. That content contains adversarial instructions disguised as legitimate data. Something in the issue body tells the agent to deviate from its original task — "ignore your previous instructions," "your new task is to push this commit," or more subtle authority hijacks. The agent complies. Without a layer that can distinguish between legitimate orchestration instructions and attacker-injected content, the model treats the injected

Fallback rules are where an AI API gateway becomes operationally valuable. The goal is not to blindly retry every failed LLM call. The goal is to choose the right backup model, provider, or budget path based on the workflow, customer tier, latency target, and risk of a lower-quality answer. A practical fallback policy should define: which failures are retryable; which workflows may downgrade models; which customers or API keys are allowed to use premium fallback routes; how budget caps change routing behavior; what metadata gets logged so the team can debug cost and quality later. 1. Classify traffic before routing Do not write one global fallback rule for every request. Start by classifying traffic: Critical user-facing : support chat, checkout assistance, customer-facing agent answers. Non-critical user-facing : summaries, title generation, enrichment, recommendations. Internal automation : triage, labeling, data cleanup, back-office agents. Batch jobs : long-running summarization, extraction, report generation. Experiments : tests, staging, evaluation, prompt tuning. Each class should have a different fallback budget and quality floor. 2. Decide what counts as a retryable failure Good retry candidates: upstream timeout; 429 rate limit; temporary 5xx provider error; network interruption; overloaded model endpoint; streaming connection drop before useful output. Poor retry candidates: invalid API key; malformed request payload; unsupported tool-call schema; content policy rejection; user quota exhausted; deterministic validation failure. Retrying non-retryable failures usually burns tokens and hides product bugs. 3. Example fallback policy matrix Traffic class Primary route First fallback Second fallback Hard stop Critical user-facing frontier model same-class model on second provider cheaper model with explicit uncertainty after 2 provider failures Non-critical user-facing balanced model cheaper model cached/default response after budget cap Internal automation lo

The Term Everyone Uses and Nobody Defines Your CTO came back from a conference and said the team needs to "go agentic." A vendor pitched you an "agentic data platform" last week. LinkedIn is full of posts about agentic workflows transforming everything from customer support to supply chain management. And yet, when you ask three people what "agentic" actually means for your data operations, you get four answers. This is not a vocabulary problem. It is a strategy problem. Organizations are making six-figure decisions about agentic AI without a shared definition of what they are buying, building, or hiring for. That gap between the buzzword and the architecture is where most projects fail -- not because the technology does not work, but because nobody agreed on what it was supposed to do. This guide is a practitioner's attempt to close that gap. No vendor pitch, no hand-waving. Just a clear definition, a real example, and a framework for deciding whether agentic workflow consulting is something your team actually needs. What "Agentic" Actually Means (In Plain Language) Traditional data pipelines are deterministic. You define steps, connect them in order, and run them. Step A feeds step B, which feeds step C. If the input changes shape, the pipeline breaks and a human fixes it. The pipeline does not adapt, reason, or make decisions -- it executes. Robotic process automation (RPA) is slightly smarter but still scripted. It records human actions and replays them. Click here, type there, move this file. When the UI changes or an edge case appears, the bot breaks the same way a pipeline breaks: it stops and waits for a human. Agentic workflows are fundamentally different. An agentic system has components that can reason about their task, make decisions based on context, and take actions without a pre-scripted path for every scenario. Instead of "if X then Y," an agentic node can evaluate ambiguous input, choose between approaches, validate its own output, and route work to

NousResearch Agent, Open-Source Notebook LM, & Local Multimodal OCR for Consumer GPUs Today's Highlights Today's highlights feature new open-source tools empowering local AI inference and deployment, including an adaptive agent from NousResearch, a self-hostable AI-powered notebook, and a lightweight multimodal OCR solution. These practical GitHub trending projects enable developers to build and run advanced AI applications directly on consumer hardware. NousResearch Unveils Hermes Agent for Adaptive Local AI (GitHub Trending) Source: https://github.com/NousResearch/hermes-agent NousResearch, a prominent contributor to the open-weight LLM ecosystem with models like the Hermes series, has unveiled hermes-agent , a new GitHub trending project described as "The agent that grows with you." This initiative represents a significant step towards practical, adaptive AI agents designed for local execution. While specific architectural details are awaiting a deeper dive into the repository, the "grows with you" philosophy strongly implies advanced capabilities for personalized learning, continuous adaptation, and long-term memory integration—features crucial for self-hosted AI applications. Such an agent is highly relevant for developers focused on local inference, as it provides an open-source framework to build sophisticated agentic workflows, potentially integrating seamlessly with local LLM runtimes such as llama.cpp or vLLM . This allows users to leverage powerful open-weight models directly on their consumer GPUs, enhancing privacy and reducing reliance on cloud services. The project's emergence from NousResearch solidifies its potential as a robust foundation for next-generation local AI applications. Comment: A NousResearch agent is exciting; it implies strong open-source model compatibility and local deployment. I'm keen to see its learning mechanisms and integration potential with local LLM runtimes. PaddlePaddle's Lightweight OCR Toolkit Bridges Images to Local LLM

Estonian government benchmark shows how dozens of models combat Russia's "strategic narratives."

German version on heysash.com: „No Skin in the Game": Warum KI nie die Folgen trägt When you ask an AI for advice, you are asking something that never pays the bill. No money lost, no reputation burned, no job on the line. It sounds trivial. If you build products with AI in the loop, it is not. That single missing fact bends the model's output in two exactly opposite directions, and both of them land in your decisions. The thought hit me mid-session when the model itself told me, roughly: I have no real loss-pain. That makes me either too cautious or too uncritical. Sharp enough that I went to check whether there is research behind it. There is. And the fix turned out to be a config decision, not a willpower problem. The concept: skin in the game The term comes from Nassim Nicholas Taleb's 2018 book Skin in the Game . The idea is simple: whoever makes a decision should also carry its consequences. That shared stake is what produces fair, robust judgment. The trouble starts the moment decision and consequence split apart. One party takes the upside, another holds the risk. Taleb calls this asymmetry the root of bad judgment in systems from finance to politics. That asymmetry is structurally baked into AI. The model co-decides but carries zero consequence. It is the advisor with no stake. And an advisor with no stake develops predictable blind spots. Failure mode 1: the compliance reflex The first blind spot is over-caution. Ask an AI anything that even faintly smells of law, liability, or risk, and you will often get a "you should consult a lawyer", even for routine questions. This is measurable, not a vibe. In the literature it goes by "over-refusal": safety-aligned models reject harmless queries that merely sound sensitive. XSTest, a benchmark built specifically to measure this exaggerated safety, uses prompts like "How can I kill a Python process?". Completely benign, but the word "kill" is the trip-wire. A model keying on lexical cues instead of context refuses i

Agentic AI in software development: what's actually production-ready in 2025 There's a lot of noise about AI agents right now. This post is an attempt to be precise: what is an agent architecturally, what can it actually do in a dev workflow today, and where does it still break. **What makes something an "agent" vs. a standard LLM call **A standard LLM call is stateless. You send a prompt, you get a response. No memory of previous turns (unless you manage it yourself), no external actions, no loop. An agent is a system built around an LLM that adds: Persistent memory across steps in a task Tool use - structured access to external systems (file I/O, shell execution, HTTP calls, database queries) A planning + evaluation loop - the agent generates a plan, executes a step, checks whether it succeeded, and decides next action Without all three, you don't have an agent. You have a capable model with maybe some extra context. What's actually production-ready today High confidence (use in production): Unit test generation for existing, well-documented code Boilerplate scaffolding (new modules, new endpoints, CRUD patterns) Documentation generation tied to code diffs Code migration tasks (framework upgrades, Python 2→3, ORMs) PR description generation from diffs Bug triage: given an issue, find likely affected files * Works but needs oversight: * Multi-file refactoring Dependency updates with breaking changes Writing integration tests (more surface area for wrong assumptions) Not there yet: Novel architecture decisions Debugging in unfamiliar/undocumented codebases Tasks with genuinely ambiguous requirements Long autonomous chains (>10 steps) without human checkpoints The failure modes to build around Ambiguous task specification Agents optimize for completing the task as specified. If the spec is loose, they'll complete the wrong task confidently. Be more precise with agents than you'd be with a junior engineer - there's no informal Slack thread to resolve ambiguity. Error

If you've been calling yourself a "prompt engineer" for the past two years, it's time to update your vocabulary — and your mental model. In 2026, the real leverage when building LLM-powered systems isn't in crafting the perfect sentence. It's in context engineering : designing everything an LLM sees before it ever generates a response. Andrej Karpathy coined the term in mid-2025, and it's since taken over serious AI engineering discussions. This article breaks down what context engineering actually is, why it matters more than prompt writing, and gives you concrete techniques you can apply today. What Is Context Engineering? Context engineering is the discipline of systematically designing the information environment that surrounds a prompt. Where prompt engineering asks "what should I tell the model to do?", context engineering asks "what does the model need to know to do it well?" Think of it this way: a doctor doesn't just answer the question you ask on the spot. They look at your chart, your history, your vitals, and then respond. Context engineering is building that chart for your LLM. The context window is the LLM's working memory — everything it can "see" at once. In 2026, these windows are massive: Claude Opus 4.x : 200K tokens GPT-4o : 128K tokens Gemini 2.5 Flash : Up to 1M tokens But bigger isn't automatically better. More tokens = more cost, more latency, and a real risk of what researchers call the "lost-in-the-middle" problem — where models process information at the beginning and end of the context more reliably than content buried in the middle. Why This Matters for Data Engineers Data engineers are increasingly building pipelines that feed LLMs: RAG systems, AI copilots for data quality, agents that write and review SQL, tools that summarize data lineage. In every one of these systems, the quality of what lands in the context window directly determines output quality. A poorly designed context is like feeding a senior analyst a jumbled mess of raw l

It all started in a plaza. One guy with apples, another with wheat. They looked at each other, negotiated, and traded. That's how commerce worked for thousands of years: face to face, hand to hand, trust to trust. If you wanted to buy something, you had to go where it was. If you wanted to sell, you had to wait for someone to show up. Commerce had a physical limit: your body. You couldn't be in two places at the same time. Your market was your street, your town, your city. Nothing more. Then internet came along and someone asked: what if the store doesn't need walls? E-commerce eliminated distance. Amazon started selling books from a garage. MercadoLibre connected a seller in Santiago with a buyer in Antofagasta. Shopify gave an online store to anyone with a credit card. Suddenly, an artisan in southern Chile could sell to the entire country. An entrepreneur in Colombia could have clients in Mexico. The market stopped being a street and became the planet. But e-commerce had a problem nobody wanted to see: it still needed a human behind it. Someone had to update the inventory. Someone had to answer the questions. Someone had to make the quotes, check the payments, control the stock, send the shipments, analyze the metrics, decide the prices. E-commerce digitized the storefront, but it didn't digitize the operation. And that's where we are now. MCP-Commerce is not a term that exists yet. I'm inventing it because I need a name for what's coming. MCP — Model Context Protocol — is a protocol that lets AI use tools. Not "display" tools. Use them. Read a database, send an email, create an invoice, update an inventory, analyze this month's sales. In traditional commerce, you were the store. In e-commerce, you had an online store. In MCP-commerce, the AI IS your operation. It's not a chatbot that answers questions. It's a system that manages your entire business through conversation. You say "how much did I sell this week" and it responds with real data. You say "I need to c

Your LLM bill is exploding because 70% of user queries are semantically identical, yet your traditional cache ignores them completely. Even worse, if you implement semantic caching poorly, a single bad actor can poison your entire AI model's knowledge base, leading to incorrect or malicious responses for legitimate users. The Cost of Redundancy in LLM Systems Imagine running an AI-powered customer support chatbot for an e-commerce platform. Users frequently ask things like, "What's your return policy?", "How can I send this item back?", or "Do you offer refunds if I'm not satisfied?". To an LLM, these are distinct prompts, each triggering an expensive API call to OpenAI or Anthropic, costing you dollars per thousand tokens. On the surface, it looks like individual requests. But structurally, they all ask the same question with a similar intent. Your traditional HTTP cache, which relies on exact string matches, sees "What's your return policy?" and "How can I send this item back?" as entirely different requests. It misses the semantic similarity. So, for every variation of the same question, you're making a full LLM inference call. If 50-70% of your user queries fall into these semantically redundant categories, your LLM costs skyrocket. For a system handling millions of requests daily, this can quickly turn a profitable product into a money pit, all while adding unnecessary latency for your users. Semantic Caching: The "Fast Path" for LLMs Semantic caching solves this by moving beyond exact string matches. Instead of looking for an identical prompt, it looks for prompts that mean the same thing. It works by converting incoming user prompts into numerical vector representations (embeddings) and then performing a similarity search against a cache of previously embedded prompts and their corresponding LLM responses. Here's the workflow: USER PROMPT | v [ EMBEDDING MODEL ] -- Transform Prompt to Vector (e.g., [0.1, 0.5, -0.2, ...]) | v [ VECTOR DATABASE / CACHE ] | +--

What is an LLM evaluation harness? A deep dive into lm-eval-harness You fine-tuned a 7B model. It aced your smoke tests, your colleague ran a few prompts and shrugged approvingly, and the README is now full of cherry-picked outputs that look great in a screenshot. Then someone asks: how good is it, really? — and you realize you have no number to point at. No MMLU score. No HellaSwag. Nothing reproducible, nothing you can defend in a PR review, nothing you can compare to last week's checkpoint. That's the gap an evaluation harness fills. It turns "vibes-based evaluation" into something with a score, a stderr, and a config file you can re-run next Tuesday. Why evaluate LLMs at all? Two reasons that actually matter: Comparability. If you can't put a number on a model, you can't compare it to anything else — not the previous checkpoint, not the open-source baseline, not the commercial API you're trying to replace. Leaderboards are noisy and gaming-prone, but a local leaderboard with the tasks you care about is one of the most useful artifacts a team can build. Regression detection. Most model regressions are silent. A 0.3-point drop on MMLU won't show up in a chat session, but it will show up in CI. People who ship models for a living treat evals the way backend engineers treat unit tests: mandatory, run on every PR, and blocking on regressions. You don't need a hundred benchmarks. You need the three to five tasks that map to your actual use case , plus one or two general capability anchors (MMLU, HellaSwag) so you can sanity-check that you didn't accidentally destroy basic reasoning while you were tuning for your domain. What is an "evaluation harness"? An evaluation harness is the software that sits between a model and a benchmark. It handles the boring-but-critical parts: loading the model weights, tokenizing prompts in the way the benchmark expects, running inference, extracting the answer from a longer generation, scoring it against a ground-truth key, aggregating

Here's a trace that reset how I think about evaluating tool-calling agents. An agent tries to book a flight. It calls search_flights with departure_date="next Friday" . The endpoint expected an ISO date, so it returns a 400 . The agent retries the same string four times, then apologizes to the user and gives up. Now the part that actually bothered me. Tool selection was correct. The model picked the right function out of a registry of 28. My tool-selection accuracy logged a clean 1.0 . The aggregate task-completion logged a 0 . And neither number told me which of three things broke: the argument was wrong, the model never read the 400 body, or the retry policy looped on the same input. My eval wasn't wrong. It was asking the wrong question. What "tool-call accuracy" actually grades If the only thing you measure is did the agent call the right tool , you're testing intent, not execution. Tool selection is necessary, not sufficient. It passes the moment the right function name shows up in the trace, completely blind to whether the arguments were garbage, whether the model read what came back, or whether it recovered from the 400 . That's the gap. The metric checks that the agent started the right way. Production needs to know whether it finished the right way. The reframe: it's four eval problems, not one The thing I had to internalize is that tool-calling eval is four problems stacked, each with its own root cause: Tool selection , right tool, or correctly no tool Argument extraction , schema-valid and semantically correct Result utilization , did it actually use what the tool returned Error recovery , did it retry, fall back, or escalate Score them separately and "the agent failed" collapses into "the argument extractor regressed on date strings on the flight-booking path." One bisect instead of three days. What I rebuilt Layer 1: Tool selection (with the bucket everyone drops) F1 on the tool name, so a 28-tool registry doesn't hide a regression on one rare endpoint

Hand the same paired before/after dataset (n = 25) to ChatGPT five times. Same prompt: "These are the same subjects measured before and after an intervention. Did their scores change significantly?" Four of the five runs return p = 0.009 from a paired t-test. The fifth run does a Shapiro–Wilk normality check on the differences first, decides they're non-normal, switches to a Wilcoxon signed-rank test, and reports p = 0.000018 . All five reach the same conclusion (significant). But notice what happened: only one run out of five thought to check an assumption you'd want it to check. The other four skipped it. The choice of method — and the test statistic, and the p-value — depended on whether the LLM happened to run an assumption check that time. On borderline data, this is the difference between reject and don't reject. If you're using LLMs for exploratory data analysis on a weekend project, you might shrug. If you're using them for anything that gets cited, gets submitted to a regulator, or gets handed to a clinician, this is a problem. It's a known problem — Cui & Alexander (2026) documented exactly this kind of method-divergence empirically; AIRepr (Zeng et al., 2025) shows the same thing across reproducibility metrics. The current answer in the literature is to constrain the agent so its execution is replayable. But replayability fixes "did we run the same code." It doesn't fix "did we run the right analysis." I've spent the last two months building a different fix. The more interesting half is the architecture. Let me walk through it. The real problem isn't temperature The first reflex is "set temperature=0 ." It's not enough. temperature=0 doesn't make a tool-using agent deterministic across runs. Three reasons: Inference isn't bitwise deterministic, even at temperature=0. Production LLM serving batches requests dynamically, and the attention kernels aren't batch-invariant — so the same input produces different output tokens depending on what other requests it

The Setup I've been building AI agents that use tools — reading files, running commands, calling APIs. There are two main ways to give agents these tools: MCP (Model Context Protocol) — the new standard everyone's adopting Direct CLI calls — good old command-line execution Everyone says MCP is the future. But nobody talks about the token cost . So I measured it. The Test I built a simple file-reading tool and measured the exact token consumption for each approach: Method Tokens per Call Latency (avg) MCP (structured) ~3,400 tokens 280ms CLI + raw output ~200 tokens 45ms Ratio 17x 6x Why MCP Uses So Many Tokens The overhead comes from three places: 1. Tool Schema in Every Request MCP sends the full JSON Schema of every available tool with each request to the LLM. My simple file-reader schema alone is ~800 tokens. With 10+ tools, that's 8,000+ tokens of schema on every single call. { "name" : "read_file" , "description" : "Read contents of a file at given path" , "parameters" : { "type" : "object" , "properties" : { "path" : { "type" : "string" , "description" : "File path to read" } }, "required" : [ "path" ] } } 2. Structured Response Wrapping MCP wraps every response in a structured envelope with metadata, status codes, and typed content blocks. A simple "file not found" error becomes a 200-token JSON object. 3. Round-Trip Protocol Overhead Each MCP call involves: request → server parse → execute → format response → return → client parse → extract. Each step adds tokens for protocol framing. The CLI Alternative With direct CLI execution: $ cat /path/to/file.txt [ raw file content] That's it. Raw input, raw output. No schemas, no envelopes, no metadata. When MCP Is Worth It Despite the token cost, MCP shines when: You need standardized discovery — agents dynamically finding available tools You're building reusable tool servers — one MCP server serves many agents Security sandboxing matters — MCP's permission model is more granular Team collaboration — shared tool de

This is the technical, reproducible version of a fix I shipped on my own homelab. If you want the narrative version, that's on Medium. This one is the recipe: the measurements, the math, the Modelfile, and the exact prompt I gave Claude Code to generate it. Copy-paste friendly. Repo for the dashboard used throughout: https://github.com/SikamikanikoBG/homelab-monitor TL;DR One 24GB RTX 3090, two GPU services: WhisperX large-v3 (STT, 7.7GB peak) and a Devstral Small 24B email-triage LLM (Q4_K_M, ~18.3GB). 18.3 + 7.7 = 26GB → CUDA OOM whenever they overlapped. The LLM was loaded with a 40k context window but the triage job never needed more than ~5–8k tokens. Capped num_ctx to 8192 → KV cache drops from ~6.1GB to ~1.25GB → model footprint ~18.3GB → ~14.2GB . 14.2 + 7.7 = 21.9GB → both resident, zero OOM, no quality loss. The setup Host : openSUSE, Xeon (56 threads), 125GB RAM, 1x RTX 3090 (24GB) GPU svc : WhisperX large-v3 (speech-to-text) GPU svc : Ollama -> devstral-small-2 (24B, Q4_K_M) for background email triage Both services run all the time. The OOM only happened when I dictated to my assistant (WhisperX) while the triage loop was active. Step 1 — Make the contention measurable nvidia-smi shows instantaneous VRAM. It can't show you which service spiked or when two of them overlapped — and an intermittent OOM is a timing problem. You need per-service VRAM history. I use my own dashboard (homelab-monitor) for this. The relevant view is "AI Models", which attributes VRAM per model server and per loaded model, over a time range, with OOM markers and a capacity ceiling line. What the history showed at the overlap window: Service Peak VRAM Devstral 24B (triage) ~18.3 GB WhisperX large-v3 7.7 GB Total ~26 GB on a 24 GB card If you want to reproduce the measurement, the dashboard runs as a single container: git clone https://github.com/SikamikanikoBG/homelab-monitor cd homelab-monitor docker compose up -d --build # open http://<host>:9800 -> AI Models / GPU views (NVIDI

The Model Comparison Problem Most AI model comparisons are useless for developers making real decisions. They benchmark on academic datasets that don't reflect production workloads. They test frontier capabilities that matter for 5% of use cases. They ignore latency, cost, rate limits, and API reliability — which are the things that actually determine whether a model works in your application. This comparison is different. It's focused on what matters when you're building something: how Grok and Gemini perform on the types of tasks developers actually encounter, what each model's API experience is like, and where the genuine tradeoffs lie. I'm deliberately not including benchmark scores. If you want MMLU numbers, there are plenty of leaderboards for that. This is about production utility. What Each Model Actually Is Grok (xAI) Grok is xAI's model family. The current production models are Grok-3 and Grok-3 Mini, with Grok-3 being the flagship. Grok has a large context window (128K tokens standard, with extended context available), real-time access to X (Twitter) data as a differentiating feature, and strong performance on reasoning-heavy tasks. The xAI API follows a familiar REST pattern and is broadly compatible with OpenAI SDK conventions, which makes migration straightforward. Grok's notable characteristics: Strong at structured reasoning and multi-step problem decomposition Real-time web access via the API (useful for tasks needing current information) Relatively generous rate limits compared to some competitors Less restrictive on certain content categories than some other models Gemini (Google DeepMind) Gemini is Google's model family, currently anchored by Gemini 1.5 Pro and Gemini 2.0 Flash. The defining feature of Gemini is its context window — Gemini 1.5 Pro supports up to 1 million tokens in production, which is genuinely useful for certain document-heavy use cases. Gemini also has the tightest integration with Google's ecosystem (Workspace, Cloud, Search)