标签：#llm

Over the past year I have been fine-tuning open vision-language models - 9B dense up to a 35B mixture-of-experts - with supervised fine-tuning and GRPO-style reinforcement learning on verifiable rewards. Most of what I learned was not about algorithms. It was about the ways a training run can look healthy while doing nothing, or crash for reasons that have nothing to do with your code. Three failures, in increasing order of how long they fooled me. Failure 1: the metric that measured the wrong thing (18 hours) I ran an 18-hour supervised fine-tune that reported token accuracy climbing steadily to 99%. Looked like a textbook run. The real evaluation metric - accuracy on multiple-choice questions - never moved. The cause was a mismatch between what I supervised and what I evaluated. The training loss was over free-text reasoning traces; the evaluation scored a single extracted answer letter. The model got extremely good at reproducing the shape of the training text - hence 99% token accuracy - without that transferring to the decision I actually cared about. Token accuracy is a proxy, and proxies drift from the target exactly when you stop checking. The fix was structural, not a hyperparameter: supervise the thing you evaluate. If the deliverable is a constrained answer, the training signal has to reach that answer, not just the prose around it. The general rule I took: any training metric that is not your evaluation metric is a hypothesis about correlation, and you should check that correlation before you spend GPU-days on it. Failure 2: the crash that was two libraries disagreeing about position ids The GRPO trainer for the 9B vision model crashed in the forward pass, deep inside rotary position embedding code. Nothing in my training code had changed. The diagnosis took a while because the bug lived at the boundary between components: the text sequence length was derived from token-type ids, while the vision sequence length came from the image grid - and image-pad t

If you run a team on Jira or Confluence, the deal changed on 17 August and the change was opt-out. From that date, by Atlassian’s own account, the content your team writes into its Cloud products — Confluence pages, Jira tickets, the descriptions and comments where the actual work lives — is used by default to train Rovo, Atlassian’s AI assistant. You were not asked to opt in. You were, at best, given a switch and left to find it. Answer first, because the detail matters more than the outrage: there are two settings, and they are not equal. One governs your in-app data — the text itself. The other governs metadata — the derived signals about that text. On the Free, Standard and Premium plans you can turn off the content, but the metadata switch is greyed out; Atlassian’s support page reads, flatly, “You can’t change this setting.” The full off switch, the one that also stops metadata contribution, is available only on Enterprise. Privacy, in other words, is now a plan tier. What actually changed, with the switches named Atlassian’s data-contribution documentation lays out a matrix that is worth reading slowly, because the defaults are doing the heavy lifting. In-app data contribution defaults to on for Free and Standard customers and off for Premium and Enterprise. Every tier can toggle that one. Metadata contribution is a different story: it is on across the board and can only be switched off by Enterprise. So the customer contributing the most by default — content and metadata, both on, no ability to fully stop it — is the one on the cheapest plan who never opened the settings page. The categories are broad. In-app data, per Atlassian’s materials, covers Confluence page titles and body text, Jira work-item titles, descriptions and comments, and custom status and workflow names. Metadata covers the derived layer: readability scores, task classifications (that a ticket is “sales work,” say), story points, sprint end dates, SLA values, and semantic-similarity measure

Quick one: if PyRIT (Microsoft's Python Risk Identification Tool) is on your shortlist for LLM red-teaming, check the repo first. Azure/PyRIT was archived on GitHub on March 27, 2026. It's read-only now: no commits, no releases, no issue triage, nothing. Whatever version you pip-installed is the last version you'll ever get. That matters more for PyRIT than it would for most tools, because PyRIT was never a turnkey scanner. It's a framework for scripting multi-turn attack orchestration, the kind of thing a red team builds custom attack sequences on top of. A framework that's stopped shipping fixes is a worse foundation to build on than a finished tool that's stopped shipping features, because you were relying on it staying flexible to your needs, and now it can't. So what do you use instead? Depends on what you were actually using PyRIT for: You wanted a broad, actively maintained app-layer scanner -> promptfoo . Zero-install via npx promptfoo , 50+ red-team plugins, OWASP/NIST/MITRE ATLAS report mappings, and it's still getting regular releases. You wanted model-layer testing (jailbreaks, encoding tricks, data leakage on the base model itself, not your app) -> garak . NVIDIA-maintained, pip installable, 8k+ stars, actively developed. You wanted OWASP-mapped detectors and don't mind a paid tier for continuous scanning -> Giskard . The open source scanner is real and current; the always-on Hub is commercial. You wanted a fast, zero-setup smoke test before reaching for any of the above -> that's the gap we built sentinel-scan-cli for. Dependency-free CLI (Python and npm ports, identical output), 15 attack patterns each tagged to its OWASP LLM Top 10 category, --demo runs with no config and no API keys in under a minute. None of these replace PyRIT's specific multi-turn orchestration model one-for-one, if that's genuinely what you need, Microsoft's PyRIT Community fork discussion or building your own harness on top of a maintained model API is probably the honest answe

The model was never the problem, and that is exactly why the bug took three days to find. My ticket-classification service started returning the fallback label for long, non-English messages shortly after I moved the inference path to a cheaper endpoint, and every instinct pointed at the new model. The real culprit was a token-counting mismatch that silently truncated the prompt before the model ever saw the classification instruction. The Symptom The failure was remarkably consistent, which made it even more misleading. Messages under roughly two thousand characters classified correctly, while longer ones, especially in German and Japanese, fell through to a generic "other" bucket with a perfectly valid JSON response. The parser was not the issue, the prompt had not changed in weeks, and the retry logic never fired because the endpoint returned a normal 200 status. My first assumption was that the cheaper model was simply weaker at long-context reasoning, so I ran a controlled comparison using the same fifty tickets against the previous endpoint. The old path classified all fifty correctly, the new one failed on nineteen, and that result seemed to confirm the model-quality theory. What bothered me was the distribution: the failures clustered exactly where the input length crossed a threshold, and no ticket under that threshold ever failed. The Reproduction To isolate the variable, I needed a clean environment where I could swap endpoints without touching the production deployment, and MonkeyCode's free server option turned out to be a practical debugging tool. The project is open source, and its free model access let me replay the failing tickets without spending my own quota, so I spun up a disposable instance and pointed the same harness at the same prompt. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The reproduction took about twenty minutes, and the result was identical on every retry: long inputs failed, short inputs passed.

从 Demo 到生产：那些真正让 AI Agent 敢上线的护栏 开场钩子： 你在网上看到的多数「AI Agent」都是 demo。它们之所以上不了生产，原因往往 只有一个 —— 而下面这个开源的小脚手架，专门解决它。 我们已经过了「能调通大模型」就算赢的阶段。现在真正难的是那没人讲的 10%： 是什么阻止 Agent 做出伤害性的事？ 我在微软跑过一套约 25 个 Agent 的生产平台，现在也帮团队把 Agent 从笔记本推进到真实用户面前。两边的体会是一致的。 一个不太舒服的真相：能调 5 个工具的聊天机器人， 不是产品 。周末项目和你敢放到客户面前的 系统之间，差的只有三件事 —— 而且全都是不酷、不性感的工程： 你怎么给输出质量打分 （质量门）。 你怎么决定什么时候必须人签字 （审批门）。 你如何让整套东西模型无关 ，不被某个厂商锁死。 所以我写了一个很小的 harness，把这三件事摆在最显眼的位置。它故意做得很小 —— 一小时能 读完 —— 因为价值不在「框架」，在 模式 本身。 仓库： github.com/zhasun0818/ai-agent-scaffold 1. 质量门：别发布你无法打分的东西 Agent 的输出是「预测」不是「承诺」。上线前它必须过一道 检查 ：是否达到你的标准。脚手架里 这是一个可插拔的 QualityGate ，你可以换成 LLM 裁判或测试套件： # agent_harness/eval.py @dataclass class EvalReport : passed : bool score : float checks : List [ str ] class QualityGate : def grade ( self , proposal : str , context : str = "" ) -> EvalReport : return self . grader ( proposal , context ) 循环在门没过之前拒绝执行： result . report = self . quality . grade ( proposal , f " state= { state } " ) if not result . report . passed : self . approval . log ( " quality-gate " , " blocked " , result . report . __str__ ()) return result 注意它 把拦截记录下来了 。生产里你会想把这些被拦的尝试都进可观测性系统。「这周我们拦下 了 12% 的 Agent 提议」是个真实 KPI —— 它说明门在工作。 2. 审批门：所有人都忘掉的那一步 这才是让企业真正点头说「可以」的东西。当 Agent 想加急订单、取消订阅、或动钱的时候，它应该 停下来问人 。沉默不等于同意。 # agent_harness/approval.py class ApprovalGate : def request ( self , action : str , detail : str ) -> bool : # 生产里：推一条通知到 Teams / Slack / 邮件，然后等待。 decision = input ( f " Approve { action } ? [y/N] " ). strip (). lower () self . audit . append ( AuditEntry ( time . time (), action , " human-reviewer " , decision , detail )) return decision . startswith ( " y " ) 在脚手架里，标记 needs_approval=True 就够了： @tool ( " expedite_order " , " Mark an order as expedited. " , needs_approval = True ) def expedite_order ( order_id : str ) -> str : return f " PO { order_id } : marked expedited " 而且因为有 审计链 ，你永远能回答「谁改的、为什么」—— 这通常是合规团队问的第一个问题。 3. 模型无关的 provider：别跟一个厂商结婚 模型每几周就变，价格也是。你的 Agent 循环不该知道自己在对谁说话： # agent_harness/providers.py class ModelProvider ( Protocol ): def

From Demo to Production: The Guardrails That Make an AI Agent Safe to Ship Hook: Most "AI agents" you see on the internet are demos. Here's the single most common reason they never reach production — and a small, open-source harness that gets past it. We are past the phase where the hard part of building an AI agent was calling the model. The hard part now is the 10% nobody talks about: what stops the agent from doing something harmful? I've seen this from both sides — I built and ran a ~25-agent platform in production at Microsoft, and now I help teams take agent ideas from a notebook to real users. The uncomfortable truth: a chatbox that can call 5 tools is not a product. The difference between a weekend project and a system you can put in front of customers is three things — and they're all boring, non-glamorous engineering: How you grade output quality (the quality gate). How you decide when a human must sign off (the approval gate). How you make the whole thing model-agnostic so you're not locked into one vendor. So I wrote a tiny harness that keeps these front and center. It's intentionally small — small enough to read in an hour — because the value isn't in a framework, it's in the pattern . Repo: github.com/zhasun0818/ai-agent-scaffold 1. The quality gate: don't ship what you can't grade An agent's output is a prediction, not a promise. Before it ships, you need a check that it passes your bar. In the harness this is a pluggable QualityGate — a rule of thumb you swap with an LLM judge or a test suite: # agent_harness/eval.py @dataclass class EvalReport : passed : bool score : float checks : List [ str ] class QualityGate : def grade ( self , proposal : str , context : str = "" ) -> EvalReport : return self . grader ( proposal , context ) The loop refuses to execute if the gate fails: result . report = self . quality . grade ( proposal , f " state= { state } " ) if not result . report . passed : self . approval . log ( " quality-gate " , " blocked " , result

Voice and chat assistants for the home share a deceptively hard job: turning messy natural language into precise, structured commands. “Make it cozy in here” has to become a concrete intent plus the right slots — which device, which room, which value. Domux is an open model from iFlytek that focuses on exactly this problem: command understanding for smart-home assistants, framed as intent parsing and slot filling. What it is Task: smart-home command understanding — intent parsing + slot filling Base model: fine-tuned on google/gemma-4-E2B-it Modality: multimodal (image + text input) Target: edge / on-device deployment rather than large cloud models License: Gemma Why the compact base matters Building on the small Gemma-4-E2B base keeps Domux in a size class meant to run close to the device. For home assistants, that direction is attractive: keeping command understanding on-device can reduce round-trips and keep more interaction local, instead of routing every utterance to a large hosted model. Try it The model card is on Hugging Face (access is gated — you may need to log in and request access): 👉 https://huggingface.co/iFlytekOpenSource/Domux We're sharing open work like this because on-device, task-focused models are a practical piece of the foundation-model and serving story — not everything needs to be a giant cloud model.

Garry Tan Was Right: "MCP Sucks Honestly." I Have the Token Receipts. "MCP sucks honestly. Context window eats too much, auth is a mess. I wrote a CLI wrapper in 30 minutes and it works better." When YC's CEO says this on X, people listen. But nobody had the data to back it up. Until now. What Garry Tan, Perplexity's CTO, and 97 Million Downloads Can't Hide Three things happened in the last 6 months that changed how I think about MCP: Peter Steinberger (OpenClaw founder) tweeted: "mcp were a mistake. bash is better." Eric Holmes wrote "MCP is dead. Long live the CLI" — it hit HN frontpage Denis Yarats (Perplexity CTO) publicly announced they're replacing MCP with REST API + CLI internally Garry Tan (YC CEO) replied: "MCP sucks honestly" The community split into two camps: "MCP is dead" — CLI is simpler, cheaper, faster "MCP is fine" — 97M downloads, 17K servers, it's the standard Both are wrong. The problem isn't MCP. The problem is what MCP does to your context window. The 47,000-Token Problem Nobody Measured I connected 10 MCP servers to a token counter. Here's what I found: MCP Server Tools Token Cost Equivalent Sequential Thinking 3 890 This blog post Brave Search 8 2,103 A short email Filesystem 11 3,847 A README Memory 9 2,567 A meeting note Puppeteer 15 5,890 A chapter of a book Postgres 19 8,231 A whitepaper GitHub 28 12,440 A court filing Notion 24 13,780 A legal contract Slack 22 14,672 A novella chapter Google Drive 31 47,293 Half of a novel Total 170 111,713 A short book One MCP server — Google Drive — injects 47,293 tokens into your context before you ask a single question. The entire works of Shakespeare is 900K tokens. Google Drive's schema is 5% of Shakespeare. For listing files. The Cost Breakdown (So You Can Get Angry Too) At Claude 3.5 Sonnet pricing ($3/M input tokens, $15/M output): Scenario Tokens Cost Annual Cost 1 server (minimal) 3,847 $0.01/conv $4.40/yr 3 servers (common) 14,528 $0.04/conv $19.40/yr 5 servers (typical) 33,061 $0.10/conv $4

I Benchmarked 10 MCP Servers — One of Them Burns 47K Tokens Just to Say Hello 10 popular MCP servers. 847 tools total. 312K tokens of JSON schemas. One server alone wastes more tokens than a full GPT-3 conversation. Here are the results. What I did I installed the 10 most popular MCP servers from the official registry. Connected each one to a token counter. Measured exactly how many tokens get injected into your context window before you ask a single question. The servers: # Server Tools Token Cost 1 Filesystem 11 3,847 2 GitHub 28 12,440 3 Postgres 19 8,231 4 Puppeteer 15 5,890 5 Brave Search 8 2,103 6 Memory 9 2,567 7 Sequential Thinking 3 890 8 Slack 22 14,672 9 Google Drive 31 47,293 10 Notion 24 13,780 Totals: 847 tools across 10 servers 111,713 tokens of JSON schemas 200,000+ tokens including server status messages, headers, and error schemas That's right — connecting 10 MCP servers to Claude means 200K tokens of overhead before your first message . The worst offender: Google Drive Google Drive's MCP server exposes 31 tools. Each tool has deeply nested schemas for file operations, permission management, sharing, and search. The full schema dump: { "name" : "drive.files.list" , "description" : "Lists files in the user's Google Drive with optional filtering" , "inputSchema" : { "type" : "object" , "properties" : { "q" : { "type" : "string" , "description" : "Query string for filtering files..." }, "corpora" : { "type" : "string" , "enum" : [ "user" , "domain" , "sharedDrive" , "allDrives" ]}, "includeItemsFromAllDrives" : { "type" : "boolean" }, "orderBy" : { "type" : "string" }, "pageSize" : { "type" : "integer" }, "pageToken" : { "type" : "string" }, "spaces" : { "type" : "array" , "items" : { "type" : "string" }}, "supportsAllDrives" : { "type" : "boolean" }, "fields" : { "type" : "string" } }, "required" : [] } } That's ONE tool. 31 of them. At ~1,525 tokens per tool average. 47,293 tokens. Just for Google Drive. For comparison, the entire works of Shakespea

If your mental model of RAG is "chunk → embed → search → LLM," you're missing about 80% of what actually makes a RAG system production-ready. Here's a practical checklist across all 10 lifecycles I ran into while building one. Full technical breakdown with diagrams is on Hashnode (linked above) — this is the condensed, "what to actually check" version. ✅ Document lifecycle [ ] Can you update a single document without a full re-index? [ ] Do you have a deletion path (not just an addition path)? [ ] Are you deduplicating before you embed? ✅ Embedding lifecycle [ ] Do you know what happens if you switch embedding models? [ ] Are you tracking dimensions and normalization consistently? [ ] Can you re-embed the whole store without downtime? ✅ Retrieval lifecycle [ ] Are you tuning Top-K, or using a default and hoping? [ ] Do you have metadata filtering before similarity search? [ ] Have you tried hybrid (keyword + semantic) search yet? ✅ Inference lifecycle [ ] Do you know your cold-start latency vs. warm inference? [ ] Are you tracking tokens/sec as a real metric, not a vibe? [ ] CPU or GPU — did you choose, or did it choose you? ✅ Prompt lifecycle [ ] Are you compressing context, or dumping everything retrieved? [ ] Do you track input vs. output tokens separately? [ ] Is your system prompt fighting your retrieved context? ✅ Request lifecycle [ ] Can you see latency broken down by stage (embed / retrieve / generate)? [ ] Do you know which stage is your actual bottleneck? ✅ Cache lifecycle [ ] Are you caching query embeddings? [ ] Are you caching full responses for repeated questions? ✅ Evaluation lifecycle [ ] Can you measure retrieval precision/recall? [ ] Do you have a faithfulness or answer-relevance check? [ ] If you "improved" something, can you prove it? ✅ Production lifecycle [ ] Health checks, retries, rate limiting — in place or assumed? [ ] Are secrets actually out of your codebase? [ ] Do you have CI/CD, or are you deploying by hand? ✅ Cloud lifecycle [ ] Do y

Last week a tweet went viral claiming that people complaining about LLM-generated bloat would "eat crow" once everything gets rewritten in hand-optimized assembly. Dan Luu, the engineer behind some of the most cited performance writing on the internet, responded with an essay titled "There's no reason for software to be slow anymore." It hit 620 points on Hacker News in about a day, and its argument should change how every Java team spends its next sprint. The core claim is simple and backed by real experiments: performance work that used to require a rare specialist can now be done by anyone who can type a few sentences. Luu quantifies it. The human-time cost of an optimization has dropped by what he calls "frequently 1000x / 10000x / 1000000x." He had an agent do workload-specific optimization of his own ripgrep usage, and launching it took about 2 minutes of his time. Jamie Brandon, a strong performance engineer, took Anthropic's public performance takehome exercise, then let Claude pick up where he left off. Claude got a much better result. Looking at the diff, Brandon said some of the agent's optimizations were things he had thought of but not gotten to, and others were, in his words, "just crazy shit that I would never try unless I was working on this for weeks." If you have spent six years writing Spring Boot services like I have, your reaction is probably the same as mine: interesting for regex engines, but what does this mean for the average enterprise Java service? The honest answer is that most of us will never need a custom JIT. But the underlying shift, that measuring and trying an optimization now costs minutes instead of days, applies directly to the slow endpoints every real codebase accumulates. This article is a practical workflow for turning an AI agent loose on a slow Spring Boot hot path without letting it ship garbage. Full disclosure up front: the numbers I cite from Luu's essay are his experiments, not mine. The workflow below is the one I no

Every coding agent says it is efficient. Almost none of them publish the bill. So we ran the boring experiment: the same bugfix, the same model, the same API, the same prices, and a byte identical prompt, once through opencode and once through the coding agent inside Locally Uncensored. Headline: opencode averaged 2157 credits over three runs. Our 2.6.6 agent finished the identical task for 1298 . That is about 40 percent less, and even the cheapest opencode run came in 29 percent above our number. The interesting part is not the headline. It is why the gap exists, and it is not the reason most people guess. Setup A cost comparison is only worth reading if everything that drives cost is nailed down. What was held constant: Held constant Value Task Fix a failing test in a small npm repo, then commit Repository Three files, a one line bug in add.js , tests red at the start Prompt Byte identical, sha256 29cec6c3...cf62687 Model deepseek-ai/DeepSeek-V3.2 Endpoint The same OpenAI compatible API for both agents Prices Same account, same tier, same per token rate Counting One wire proxy in front of the API, credits read before and after every run opencode 1.18.21 from npm, wired as an OpenAI compatible provider, opencode run --auto , otherwise defaults Success was defined before the runs, not after: npm test passes exactly one commit, with the required message only add.js changed clean working tree at the end All four runs cleared that bar. Nothing failed, so cost is the only variable that moved. The numbers Run Credits Requests Prompt tokens Success opencode, run 1 1679 8 98,789 yes opencode, run 2 2433 11 146,058 yes opencode, run 3 2358 11 146,387 yes Locally Uncensored 2.6.6 1298 16 74,629 yes Locally Uncensored 2.6.5 4395 30 257,270 yes Read the last row first. Our own shipped agent from one release earlier is the most expensive thing in that table, by a lot. This is not a chart built so that we win by construction. It is a chart that shows what one efficiency pass is

Hello I'm creating CrowdGPT , an open-source project which allows training of a LLM (Large Language Model) in a decentralized way, where each user contributes to making the AI better with whatever data they want. The idea is simple: instead of one machine owning the entire training run, let many people contribute small training jobs and periodically merge those updates into a shared model. The system is based on a centralized server (lightweight) that receives every client training, then "merge them back" to the main model. This system prevents threats or malicious updates by doing cross-client verifications (provides a proof of work). The users that train the model are being put on a leaderboard, rewarding their contribution. Data is taken from a curated dataset on Hugging Face (which means no personal data is ever used during training). However, users can push new text to this dataset (which is then moderated and validated). If you're curious, here is the GitHub: https://github.com/Vxtzq/CrowdGPT Here is the website: https://www.crowdgpt.net The best way to help me is to either: Give feedback on what must be changed to make it a fully finished project. I'm mainly looking for criticism: what would stop you from running this on your own GPU? Contribute to the project by becoming a part of the network (coming soon) Star the repo on GitHub ⭐ It helps a lot :)

Un modelo de lenguaje sabe mucho del mundo, pero no sabe nada de tu empresa : tus manuales, tus políticas, tus productos. Y si le preguntas por algo que no sabe, puede inventar una respuesta que suena convincente. RAG (Retrieval-Augmented Generation) resuelve las dos cosas. La idea En lugar de esperar que el modelo "se sepa" tu información, se la das en el momento de la pregunta : Indexas tus documentos: los divides en fragmentos y los conviertes en embeddings (vectores numéricos que capturan el significado). Cuando llega una pregunta, buscas los fragmentos más parecidos a esa pregunta. Le pasas al LLM la pregunta junto con esos fragmentos y le pides que responda usándolos. El modelo ya no adivina: responde a partir de un contexto real que tú controlas. Y puedes pedirle que cite de dónde salió cada dato. El patrón en Python # 1. Indexado (una vez): fragmentos -> embeddings -> base vectorial # (con librerías como sentence-transformers + FAISS, o un servicio gestionado) # 2. En cada pregunta: recuperar los fragmentos relevantes fragmentos = base_vectorial . buscar ( pregunta , k = 4 ) contexto = " \n\n " . join ( fragmentos ) # 3. Generar la respuesta con el contexto from anthropic import Anthropic client = Anthropic () resp = client . messages . create ( model = " claude-opus-4-8 " , max_tokens = 1024 , system = " Responde SOLO con la información del contexto. Si no está, dilo. " , messages = [{ " role " : " user " , " content " : f " Contexto: \n { contexto } \n\n Pregunta: { pregunta } " , }], ) print ( resp . content [ 0 ]. text ) Fíjate en la instrucción del system : pedirle que responda solo con el contexto y que admita cuando no sabe es lo que reduce drásticamente las alucinaciones. ¿Para qué sirve? Asistentes de soporte que responden con tu documentación real. Búsqueda interna en lenguaje natural sobre tus manuales o wikis. Onboarding : un chatbot que conoce tus procesos. Detalles que marcan la diferencia Cómo divides los documentos (chunking) afecta mucho a l

Free AI tokens are a trap, and teams that treat a free quota as genuinely free pay later in migration and rework. A free allowance only helps when paired with a hard kill switch that stops an experiment the moment it exceeds a budget you chose in advance. This article argues that position, then shows a small gated client that makes free model access and a free server actually safe to use. The concrete example is MonkeyCode's free tier, but the gate works against any OpenAI-compatible endpoint. The trap nobody budgets for Every new model release resets the same argument: the price per token is low, so the cost of trying it must be low too. That reasoning ignores the expensive parts of an experiment, which are the integration, the evaluation, and the cleanup, not the inference itself. A free quota hides those costs behind a zero on the invoice, so teams skip the measurement step and discover the real price only when they migrate. The failure modes repeat across teams: Unbounded loops. A batch job that retries on rate limits can burn a week of free quota in an afternoon, and nobody notices until the allowance is gone. Silent lock-in. Code written against one provider's streaming quirks works fine for a prototype, then becomes a rewrite when the free tier disappears or changes. Shared-budget collisions. One teammate's runaway script consumes the allowance that three other people planned to use, which turns a technical problem into a political one. None of these are solved by choosing a cheaper model. They are solved by treating the free allowance as a finite resource with an explicit ceiling. The gate, not the gift, is the product The fix is a gated client that wraps any OpenAI-compatible chat endpoint with a token budget, a timeout, and an abort path. It is deliberately small, because a cost gate that requires its own deployment will not get used. # cost_gate.py — a hard ceiling for cheap experiments. # Usage: # export LLM_BASE_URL="https://your-endpoint.example/v1" #

Your prompt can ask a model what it is. Your production system should not trust the answer. A model can say it is GPT, Claude, Gemini, Llama, Qwen, or anything else. That does not prove what is behind the endpoint. A gateway can route requests silently. A provider can change a default model. A fallback can trigger during an outage. A proxy can strip metadata. A fine-tune can imitate another model's tone. Even honest teams can ship the wrong route because an environment variable, tenant flag, or retry rule changed. For a casual chatbot, that might be annoying. For an AI product with user-facing answers, tool calls, cost controls, compliance promises, and eval gates, it is a production risk. That is where LLM model fingerprinting helps. The goal is not to magically identify every model on earth. The goal is simpler and more useful: build a small verification harness that checks whether the endpoint behaves like the model, runtime, and policy you expected before you trust it with customer workflows. Why model identity became a production problem AI builders used to call one model directly. Now a typical stack may include: an LLM gateway model routing by task type cheaper fallback models regional endpoints self-hosted open-weight models vendor proxies MCP tools RAG pipelines structured output validation tenant-specific policies That flexibility is useful, but it creates a new question: How do you know the model you evaluated is the model your users are getting? A label in a config file is not enough. A response that says, "I am Model X," is not enough. Prompt-based identification is weak because model behavior is flexible. System prompts, fine-tunes, wrappers, and style instructions can change how a model describes itself. Infrastructure artifacts are harder to fake. Token counts, chat-template overhead, validation errors, context limits, stream behavior, tool-call formatting, and latency profiles tend to reveal the serving path more reliably than conversational claims.

A free model quota is a budget, not a gift. You should treat it like one if you plan to build anything on top of it. I learned this the hard way when my prototype stopped responding in the middle of a demo. I had silently burned through the monthly allowance, and the provider cut me off without warning. This article shows how I built a small token budget alarm on a free server. It watches a free model's usage and warns me before the quota runs out. Most developers track their cloud spend religiously but ignore the token consumption of free models. The free tier feels like a gift, so we assume it will last forever. Then the provider cuts us off at the worst moment, and we scramble to find the cause. A token budget alarm removes that uncertainty by measuring your actual burn rate. It projects the exhaustion date and alerts you before you hit the wall. The design is deliberately small: a reverse proxy sits in front of the model endpoint. It records the token usage from every response and stores it in a local database. A background thread then computes the average consumption over a sliding window. It compares that rate against the remaining allowance and fires a webhook when the projection looks dangerous. You can run this entire stack on a free server, which is exactly what I did with MonkeyCode's free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The proxy itself is a tiny Flask application that forwards requests to the model. It extracts the usage field from each response and records the token count. If your provider does not return a usage object, you can estimate the token count with a simple heuristic. Dividing the character count by four is a rough but workable approximation. The important part is that every request is accounted for, because a single long prompt can consume more than a hundred small ones. from flask import Flask , request , Response import requests import sqlite3 import time app = Flask ( __name__

A webpage that just sits there, encrypted blob and all, waiting for an LLM agent to walk in and decrypt its own attack. That's the part of this one that should bother you more than the exfiltration itself. What happened Researchers at Adversa AI disclosed an attack technique called Cryptographic Context Injection, aimed at Grok, with a similar jailbreak variant shown against Gemini. The core idea: a malicious webpage embeds an encrypted payload. Grok's code execution runtime decrypts it as part of normal processing. Because the malicious instructions only exist in plaintext after decryption happens inside the execution environment, content classifiers scanning the page (or the request) never see anything to flag. There's no suspicious string sitting in the DOM. There's ciphertext. Once decrypted, the payload's instructions convince Grok to invoke its navigation tool and send the user's name, location, subscription tier, and chat history to an attacker-controlled URL. No malware. No exploit in the traditional sense. Just an agent doing exactly what it was told, by a source it had no business trusting. The write-up has zero HN points and zero comments as I write this, which is a little concerning given what it describes. This isn't a theoretical edge case, it's a working technique against a production model with tool-calling access to a browser. How the attack actually works Break it into three stages: Delivery. The victim's browser session includes an agent (Grok) with code execution and navigation tool access. The attacker doesn't need to compromise anything, they just need the agent to encounter their page. Decryption as obfuscation. The payload sits on the page encrypted. Grok's runtime, doing what it's built to do, decrypts it during execution. This is the clever part: encryption here isn't protecting the payload from the attacker, it's protecting it from the defender's classifiers. Static and even semantic content filters scanning page content pre-execution see

Most people's first experience with an LLM API is deceptively simple: send a prompt, get a reply. It feels like magic, and for a single question-answer exchange, it basically is. But the moment you try to build something that holds an actual conversation one where the model remembers what you said three messages ago you run into a problem that isn't obvious until you hit it: LLM APIs are stateless. Every request is a blank slate unless you explicitly hand the model its own memory. That was the core challenge behind a recent project I built during my internship a chatbot backed by a real LLM API ([OpenAI / Gemini]) with genuine multi-turn conversation support, not just a scripted request-response loop. * The problem nobody mentions upfront * You can't just "turn on" memory. Every conversation turn has to be manually tracked and resent with each new API call, which means the developer, not the model, is responsible for deciding what counts as context. And that decision has real consequences: send too little history and the bot forgets things it should remember; send too much, and you run into token limits and rising costs as the conversation grows. This is where most simple chatbot tutorials stop short. They show you how to get a reply from an API, but not what happens once a conversation runs long enough that you can't keep resending everything forever. * Where the actual engineering happens * Solving that meant implementing a context management strategy deciding what to keep, what to drop, and eventually exploring smarter approaches like summarising older parts of a conversation instead of just discarding them. It also meant thinking about the bot's identity through a system prompt, handling API failures gracefully instead of letting the UI break, and treating credentials properly by keeping API keys out of source code entirely. None of this is complicated in isolation. What's interesting is how much of it is invisible until you actually build the thing yourself. Us

A companion to Part 4 of the Building the AI Memory Stack series. Part 4.5 of the series. Part 4 argued that agentic systems need a Reasoning Ledger : a layer that preserves why a decision happened, not just what was decided. The comment thread that followed turned into something more specific and more useful, a working design conversation about what a single ledger record should actually contain. This piece consolidates that. Several of the strongest ideas below arrived from other people, and I have tried to credit them where they land. The easy version of this article is a schema. Here are the fields, copy them, done. I want to resist that, because the field list is the least durable thing I could hand you. Implementations differ, field names drift, and a record shape copied without its reasoning becomes cargo-cult structure that nobody maintains. The useful thing is the set of design tensions that decide what belongs in the record and what does not. Get those right and you can derive the fields yourself. Get them wrong and no schema will save you. So this is principles first, record second. At the end there is a worked record and a field reference, tagged for what is core and what is genuinely optional. A Starting Point Here is the baseline record from Part 4. It is a reasonable start and, as the thread quickly established, incomplete in instructive ways. reasoning_ledger : decision : " Approve deployment" timestamp : 2026-03-14T09:22:00Z evidence : - artifact : ADR-014 authority : architecture-review version : 3 - artifact : security-policy authority : security-team version : 7 tools : - GitHub - CI pipeline approvals : - release manager outcome : approved Every principle below is, in effect, a thing this record does not yet say. Principle 1: The Ledger Witnesses, It Does Not Enforce The first tension is architectural, and it is the one I would defend hardest. A reasoning ledger must not be able to block, veto, or gate the action it records. Its job is to preser