标签：#llm

Honestly it's not a new concept. this feature already existed in models before. problem was the models were just weak. Looping only works if each attempt gets the agent closer to the correct solution. Earlier models weren't consistent enough for that. They often misunderstood feedback, repeated the same mistakes, or got stuck in an infinite loop. Instead of improving with each iteration, they frequently failed to make meaningful progress, eventually consuming large numbers of tokens without solving the problem. The Context Window Limitation Earlier language models had much smaller context windows. As the agent went through more iterations, the conversation history and reasoning gradually filled the available context. Once the context window was exceeded, older messages had to be dropped or compressed into summaries. As a result, the agent could forget previous failed attempts, lose important clues or reasoning, and sometimes repeat the same mistakes it had already made. So what did modern models actually fix? Bigger context windows Models can now hold way more of the conversation/history without forgetting, so the agent doesn't need to spin up a fresh session every few iterations. it can just keep looping with the full history of what failed and why. modern models also got way more consistent earlier if you asked a model to fix the same bug 5 times you'd get 5 different half-baked answers, now it actually converges toward the real fix. and tool use got better too . Old models could write code but couldn't run it and read the actual error, now they call a test runner, see the real failure, and fix that exact thing which is literally what makes the "verify" step possible. And then there's inference it is simply the process of a model generating an answer. like when you type "write a java binary search," the model reads your prompt, thinks, and generates code that whole process is inference. every time the model generates text, that's one inference. now here's the thin

In Part 1 , we learned that building great AI applications isn't just about writing better prompts. It's about providing the right context . But that naturally leads to another question: How much context can an AI actually understand? If you've used ChatGPT, Claude, Gemini, Cursor or any AI coding assistant for a while, you've probably experienced something like this. 🤔 "Didn't I Already Tell You That?" Imagine you're debugging a production issue with an AI assistant. You start by explaining the architecture. Then you share the API flow. Then database schema. Then logs. Then stack traces. After 30 minutes of conversation, you ask: "So what's causing the bug?" Instead of giving the answer, the AI responds: "Could you share your database schema?" You stare at the screen. "I already did..." Sometimes it even forgets details from earlier in the same conversation. Naturally, people assume: The AI has bad memory. The model is unreliable. The conversation is broken. In reality, something completely different is happening. You're running into one of the most important concepts in modern AI systems: The Context Window. Understanding this concept changes how you interact with AI—and more importantly, how you build AI-powered applications. 🧠 Before We Talk About Context Windows… We first need to understand something much smaller. Tokens. Almost every AI provider mentions them. Pricing is based on them. Context windows are measured using them. Yet many developers still think: 1 token = 1 word That isn't true. 🔤 What Exactly Is a Token? A token is the basic unit of text that an AI model processes. Humans naturally read text as: Characters Words Sentences Large Language Models don't. Before text reaches the model, it's converted into smaller pieces called tokens by a tokenizer. The model never sees your original sentence. It only sees a sequence of tokens. Think of a tokenizer as a translator between humans and AI. You write English ↓ Tokenizer ↓ Sequence of Tokens ↓ LLM The toke

Back when we introduced MCP support , we ended on a teaser: Phase 3 would tackle Sampling —letting servers request completions from the host instead of only exposing tools and resources to it. NodeLLM 1.17 delivers on that, and pairs it with a second, unrelated but overdue improvement: precise control over how tool calls execute, now available consistently in both core and the ORM persistence layer. 🔄 MCP Sampling: Closing the Loop Sampling inverts the usual MCP direction. Instead of the client asking the server for tools, the server asks the client to run an LLM completion on its behalf. This lets an MCP server offer LLM-powered capabilities—summarization, classification, drafting—without needing its own API key or provider integration. createLLMSamplingHandler answers those requests using a real NodeLLM instance, so a server's tool ends up powered by whatever model you configure client-side: import { createLLM } from " @node-llm/core " ; import { MCP , createLLMSamplingHandler } from " @node-llm/mcp " ; const llm = createLLM ({ provider : " openai " }); const mcp = await MCP . connect ( { command : " node " , args : [ " ./sampling-server.mjs " ] }, { sampling : createLLMSamplingHandler ( llm , " gpt-4o-mini " ) } ); const tools = await mcp . discoverTools (); // The server only advertises sampling-backed tools once it sees // the client declared sampling support during the handshake. If you need full control over how a sampling request is answered—routing by model hint, injecting your own guardrails—pass a plain handler function instead of { llm, model } . It receives the raw sampling/createMessage params and returns a CreateMessageResult , so you decide exactly how (or whether) to answer. ⚡ Concurrent Tool Execution When a model returns several independent tool calls in the same turn, NodeLLM has always executed them one at a time. That's safe by default, but wastes time when the calls don't depend on each other—three weather lookups for three different cities, s

Fable and Opus may not be the most comfortable tools for engineers who learned to code by hand. I started thinking about this after reading Simon Willison's recent note . His point is simple: with a strong coding agent like Fable, it may be better to let the model exercise its own judgment than to spell out every condition yourself. Instead of writing detailed rules like "run tests for larger features, but not for small copy changes, except for design changes...," you can simply say: write and run tests where appropriate. The same applies to cost. Rather than deciding manually which tasks should go to which model, you can ask the agent to choose an appropriate lower-cost model and delegate the work to a subagent. Manual cars and automatics This is a rough analogy, but it feels similar to driving a car. People who enjoy driving often like manual cars. They want to choose the gear themselves. They want to feel the engine speed and have the car respond directly to their intent. For people who simply want to get somewhere, an automatic is easier. Software engineers are similar. If you have written code professionally for a long time, you usually have your own way of working. You may want to get the types right first. You may prefer small diffs. You may have a specific sense for how granular tests should be. You may even have an order in which you like to read an unfamiliar codebase. (At least, I hope you do.) For someone with that kind of style, a highly autonomous model like Fable or Opus can feel a little too automatic. The stronger the model, the more small instructions get in the way This is the same structure as management in human organizations. A junior member needs concrete instructions: read this document from this angle and summarize it in this format. A senior member can take a rougher assignment: I want to solve this problem, so investigate it, come up with an implementation plan, and move it forward. Of course this does not mean throwing work over the wall.

What We Built in This Guide In the previous guide, we went from RAG to cloud deployment. In this guide, we systematically implemented everything needed to take that system to production . evals/ dataset.py # Evaluation dataset eval_rag.py # Context Recall · Relevancy · Faithfulness observability/ traced_rag.py # RAG pipeline tracing with @observe() (Langfuse v4) traced_agent.py # Trace each Agent step security/ input_validator.py # Prompt injection detection output_validator.py # PII masking and leakage detection guardrails.py # Rate limiting, security log integration secure_rag.py # RAG with guardrails llmops/ prompt_registry.py # Prompt version management (v1.0–v1.2) ci_eval.py # Quality gate (Overall ≥ 75% to deploy) cost_tracker.py # API cost tracking finetuning/ prepare_dataset.py # Convert to Alpaca format train_lora.py # LoRA fine-tuning (r=8, 2 min on CPU) inference.py # Compare with base model multiagent/ search_worker.py # Search specialist worker quality_worker.py # Quality check specialist worker orchestrator.py # Task decomposition and result integration 14_multiagent.py # Execution script governance/ ai_registry.py # AI system inventory risk_assessor.py # Risk assessment (score 0.18 → LOW) audit_logger.py # Audit log (Article 12 compliant) compliant_rag.py # RAG with AI disclosure (Article 50 compliant) Key Design Decisions from Each Chapter Chapter 2: Evals Combining rule-based (Context Recall, Answer Relevancy) with LLM-as-a-Judge (Faithfulness) strikes the right balance between speed, cost, and coverage. Chapter 3: Observability (Langfuse v4) Adding @observe() decorators is all it takes to start recording traces. The critical v4 change: you must call get_client() after load_dotenv() . Chapter 4: Security Defense in Depth is the principle: Input validation → System prompt → Output validation → Rate limiting — four layers of protection. Chapter 5: MLOps / LLMOps On every push to GitHub, Evals run automatically. Only when the quality threshold (Overall

Introduction Through Chapter 7 (Multi-Agent) , we have a complete, functioning AI system. The final step is building organizational infrastructure to operate AI safely over time. [Before] Technical safety Security → Block malicious input Evals → Measure quality [Now] Organizational / regulatory safety Governance → Know what AI systems are in use Risk mgmt → Classify and assess risks Audit logs → Record who did what, when EU AI Act → Regulatory compliance EU AI Act Status (as of June 2026) The EU AI Act came into force on August 1, 2024, with full enforcement on August 2, 2026 . Transparency rules (disclosing when users are interacting with AI, labeling AI-generated content) also take effect on that date. AI systems are classified into three risk tiers: Risk Level Description Examples Prohibited Not permitted Social scoring, manipulative AI High Risk Strict regulation Hiring, credit scoring, law enforcement Limited Risk Transparency obligations Chatbots, AI-generated content Minimal Risk No regulation Spam filters, game AI Our RAG system's classification: Limited Risk (chatbot). We are required to disclose to users that they are interacting with AI. Directory Structure pgvector-tutorial/ ├── existing files └── governance/ ├── ai_registry.py # ★ AI system inventory ├── risk_assessor.py # ★ Risk assessment ├── audit_logger.py # ★ Audit logging └── compliant_rag.py # ★ Governance-compliant RAG 1. AI System Inventory — governance/ai_registry.py Most organizations lack a systematic inventory of their AI systems, making risk classification and compliance planning difficult. Knowing what you have is the essential first step. # governance/ai_registry.py """ AI system inventory Centrally manage all AI systems in use across the organization. Forms the foundation for technical documentation required by EU AI Act Annex IV. """ from datetime import datetime from dataclasses import dataclass , asdict from enum import Enum class RiskLevel ( Enum ): UNACCEPTABLE = " prohibited " HIG

TL;DR— Most production AI teams build safety layers using the content-moderation mental model: classify input, classify output, block or pass. But the incidents that actually take down AI systems in production look like distributed-systems failures— retries amplifying bad state, cascading errors across agent steps, silent drift with no rollback path. Guardrails need to borrow from SRE, not from trust-and-safety. Ask a team how they handle AI safety in production and you'll get the same answer almost every time: an input classifier, an output classifier, maybe a moderation API bolted on the side. This is the content-moderation mental model— filter bad stuff in, filter bad stuff out. It's borrowed wholesale from trust-and-safety teams who spent a decade building spam filters and abuse detectors. It's also the wrong model for most of what actually breaks AI systems in production. The incidents that page you at 2am rarely look like a jailbreak slipping past a classifier. They look like distributed-systems failures: a retry loop that amplifies a bad tool call, a hallucinated intermediate result that poisons every downstream step, a silent shift in output distribution that nobody notices until a customer complains three weeks later. These are not content problems. They're systems problems, and they need systems solutions. The Cascade, Not the Jailbreak Consider a typical agent pipeline: retrieve context, call a model to plan, call tools, call a model again to synthesize, maybe loop if a tool fails. Each step has some non-zero error rate. In a single-call chatbot, that error rate is the whole risk surface. In a five-step agent chain, errors compound, and worse, they compound non-linearly because failed steps often trigger retries, and retries on a stateful action are not free. A model that hallucinates a tool argument doesn't just produce one bad output— it produces a bad state that the next step reasons over as if it were true. If that next step is another LLM call, it wi

Semantic coherence is not a quality metric or an alignment outcome. It is the structural condition that determines whether meaning remains stable, interpretable, and legitimate as the system accelerates. In the broader architecture of sovereign AI, semantic coherence is the component that ensures meaning does not fragment under pressure. Semantic coherence is the difference between a system that understands meaning and a system that merely produces plausible output. The Perception Semantic coherence is often treated as a linguistic property: clarity, consistency, interpretability, explainability, or “staying on topic.” In this perception, coherence is something evaluated externally — a measure of how well the system’s outputs align with human expectations. This view assumes coherence is a surface behaviour: does the output make sense does it follow logically does it stay within context does it appear consistent But this perception is fundamentally flawed. It treats coherence as an effect rather than a structural property. When coherence is treated as external, it becomes subjective, fragile, and easily destabilised by acceleration. The Reality Semantic coherence is not external to the system. Semantic coherence is the system. A system is coherent when its meaning remains stable across: acceleration optimisation pressure boundary transitions external inputs internal state changes If the architecture cannot maintain coherence internally, then: meaning fragments behaviour becomes inconsistent transitions lose legitimacy boundaries collapse under pressure governance becomes interpretive A system without semantic coherence does not understand meaning. It performs meaning. Semantic coherence is not about producing sensible output. It is about being structurally incapable of semantic drift. What Semantic Coherence Actually Is In sovereign AI, semantic coherence is the architectural logic that ensures: meaning remains stable under acceleration semantics remain consistent ac

Book: Observability for LLM Applications — Tracing, Evals, and Shipping AI You Can Trust Also by me: Agents in Production — the companion book in The AI Engineer's Library (2-book series) My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools Me: xgabriel.com | GitHub Your triage agent has been in production for three months. The traces look clean. Every span is green, every run terminates, the p95 latency is flat, the token bill is boring. Then support forwards you a screenshot: the agent routed a billing refund to the security queue. You pull the trajectory. Nothing is broken. The agent called a reasonable tool with reasonable arguments, got a reasonable result, and picked the wrong queue with total confidence. That is silent drift. The trace shows you what the agent did. It does not tell you whether what it did was any good. Between a model provider's minor version bump, a prompt tweak someone shipped on Tuesday, and the slow shift in your incoming traffic, the quality of your agent's decisions moves. It rarely announces itself with an error. It shows up as a support ticket, then a second one, then a churned account. You catch it the same way you catch a memory leak: with a baseline and an alarm, not by staring at dashboards. Decision quality is a distribution, not a number The failing traces in Chapter 12 of Agents in Production are the easy case. Twenty retries of the same empty search is visibly wrong. You see the loop count in the invoke_agent parent and you know. Most quality regressions are not visible in a single trace. They show up only when you look at the distribution of decisions across thousands of runs. So the first thing to instrument is the decision itself. If you followed the tracing chapter you already emit a small fixed vocabulary per chat span: span . set_attribute ( " gen_ai.agent.step " , 3 ) span . set_attribute ( " gen_ai.agent.decision " , " call_tool " ) span . set_attribute ( " gen_ai.

Book: Agents in Production — Building, Tracing, and Shipping Multi-Step AI You Can Trust Also by me: Observability for LLM Applications — the companion book in The AI Engineer's Library (2-book series) My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools Me: xgabriel.com | GitHub In June 2025 two engineering teams posted opposite advice in the same week. Anthropic shipped How we built our multi-agent research system : an orchestrator dispatching subtasks to worker agents, beating a single agent by 90.2% on breadth-first research. A few days later Cognition, the team behind Devin, published Don't Build Multi-Agents , arguing that parallel subagents without shared context produce fragile systems. Both were right. They were describing different workloads. Anthropic's research agent is embarrassingly parallel: four workers go read four things and come back with four small summaries. Cognition's target is writing code, where every edit depends on every other edit and context cannot be sliced. Most people get the plumbing wrong, not the decision. Once you have two agents that need to coordinate, you have to choose how they talk. That choice decides your failure modes long before the models do. Handoffs vs a shared bus There are two ways to wire agents together, and they fail differently. A handoff transfers control. Agent A finishes, hands the whole conversation to Agent B, and steps out. This reads well in a demo. In production it means the transcript grows on every hop, and by the fourth agent you are paying to re-read a conversation nobody trimmed. Handoffs also lose the parent: once A hands off to B, nobody is holding the original task to check the final answer against it. A shared bus keeps a supervisor in charge. Workers never talk to each other. They receive a small typed task, do the work, and return a small typed artifact to the supervisor, which composes the result. This is the shape of Anthropic's research

Book: Agents in Production — Building, Tracing, and Shipping Multi-Step AI You Can Trust Also by me: Observability for LLM Applications — the companion book in The AI Engineer's Library (2-book series) My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools Me: xgabriel.com | GitHub The agent works on your laptop. It passes evals. Your manager asks when it ships and you say Monday, because the modeling is done. Then you try to put it behind a load balancer and it falls apart, because you deployed it like a web service. An agent is not a web service. A web service answers in milliseconds and forgets. An agent thinks for minutes, burns tokens across two or three providers, streams partial output to a browser, and sometimes decides to call delete_invoice on the eighth turn. Every deployment decision you make flows from one question: what does this thing do to your infrastructure while it is running? Here is how to package it, where to hold state, and how to scale a workload whose bottleneck is a model call you do not control. The shape is decided by the longest step The single rule that saves you the most pain: an agent's deployment shape is decided by its longest step, not its average step. A support chatbot answers in two seconds. A code-review agent thinks for six minutes. A research agent runs for forty. You cannot put all three behind the same HTTP endpoint and expect any of them to survive. Pick the pattern that matches the longest step, then cap the rest with timeouts. Under 30s → stateless HTTP endpoint (Cloud Run, Fly.io). 30s to 5m with a user watching → streaming over WebSocket or SSE. 5m to an hour, async → queue plus worker (Temporal, Inngest, or Redis). Longer than an hour → still queue plus worker, whether you like it or not. Do not hold an HTTP request open for forty minutes. Something you did not know existed will kill it at the worst moment: a proxy, a CDN, a load-balancer idle timeout. Package it: p

Book: Agents in Production — Building, Tracing, and Shipping Multi-Step AI You Can Trust Also by me: Observability for LLM Applications — the companion book in The AI Engineer's Library (2-book series) My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools Me: xgabriel.com | GitHub On April 2, 2026, Microsoft shipped agent-framework 1.0 and, in the same blog post, moved AutoGen into maintenance mode. Semantic Kernel went with it. Three overlapping projects folded into one package with stable APIs and long-term support. Microsoft framed the move as a consolidation. If you had an AutoGen project that morning, you woke up with a migration. That is the shape of this whole category. The framework landscape you pick from today is not the one you picked from a year ago, and it will not be the one you pick from next year. So the useful question is not "which framework is best." It is "which framework has which wedge, and which trade-off comes with it." Here is an honest read on six frameworks worth installing in 2026, and when to reach for each. The churn is the feature, not the bug Before the tour, one thing that changed the math: the wire formats underneath these SDKs converged. Every framework here speaks MCP for tools. Most support A2A for cross-framework handoffs. Model Context Protocol started as an Anthropic proposal at the start of 2025 and is now the default way agents pick up external tools. That convergence means the framework you pick locks you in less than it used to. You are still locked at the abstraction layer, though. Migrating a production system from CrewAI to Pydantic AI is a rewrite of every Agent definition and every tool decorator. The pick is sticky. Choose it with that in mind. LangGraph : durability as the wedge Reach for LangGraph when your agent has to survive a crash. It models the agent as a graph with checkpointers backed by Postgres or SQLite, so a workflow that dies at step seven resumes a

Book: Agents in Production — Building, Tracing, and Shipping Multi-Step AI You Can Trust Also by me: Observability for LLM Applications — the companion book in The AI Engineer's Library (2-book series) My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools Me: xgabriel.com | GitHub You ship an agent that resolves billing disputes. It works in the demo. Two weeks later a support ticket lands: the agent tried to refund $4,000 on a $19 charge. You read the trace. The model returned a JSON blob, your code did json.loads , pulled amount , and passed it straight to the payments API. No cap. No type. No check. The model hallucinated a number and your code trusted it. The model is stochastic. Your code does not have to be. The gap between those two facts is where most production agent bugs live, and it is exactly the gap Pydantic AI is built to close. The wedge is types Most agent frameworks hand you an Agent object and a bag of strings. Pydantic AI hands you Agent[Deps, Output] — a generic parameterized by its dependency type and its output type. The IDE and your type checker read those parameters. So does the runtime. Install pulls in the framework plus an optional tracing extra: pip install "pydantic-ai[logfire]" The smallest program that earns its keep: from dataclasses import dataclass from pydantic import BaseModel from pydantic_ai import Agent , RunContext @dataclass class Deps : customer_name : str class SupportReply ( BaseModel ): reply : str escalate : bool agent = Agent ( " anthropic:claude-opus-4-8 " , deps_type = Deps , output_type = SupportReply , system_prompt = " You are a support agent. " , ) A tool is a plain function whose type hints become the schema the model sees, and the run returns the validated SupportReply : @agent.tool def customer_name ( ctx : RunContext [ Deps ]) -> str : return ctx . deps . customer_name result = agent . run_sync ( " What is my name? " , deps = Deps ( customer_name = " Ana "

The Problem That Wouldn't Leave Me Alone Pakistan has 220 million people. A functioning legal system. Hundreds of Acts, ordinances, and constitutional provisions that technically protect every citizen. Almost nobody can use them. The median lawyer's consultation fee in Karachi is more than what many families earn in a week. Legal aid is understaffed and geographically concentrated in major cities. And the laws themselves? Written in English — a language most of the population reads functionally at best, and doesn't speak at home at all. So when a landlord illegally locks someone out. When a factory worker gets fired without severance. When a woman wants to know her inheritance rights. When a tenant needs to understand what "Section 16 of the Rent Restriction Ordinance" actually means for their specific situation — they either find a lawyer they can't afford, ask someone who doesn't really know, or quietly give up. This isn't a knowledge problem. It's an access problem. I'm a CS student at Sukkur IBA University in interior Sindh — not Karachi, not Islamabad. The kind of city where you feel the gap between what the law says and what people actually know it says every single day. That gap is where HAQ started. HAQ is an Arabic and Urdu word. It means right — as in, what is rightfully yours. The name felt important. The Core Idea: Ask the Law, Get the Law There's a specific failure mode with AI and legal questions that drove every design decision I made, and it's worth naming clearly. Standard LLMs — any of them — will answer legal questions confidently. They'll cite "Section 144" or "the Transfer of Property Act" with total authority. They are often wrong. Sometimes subtly: the section exists but doesn't say what the model claims. Sometimes obviously: the Act doesn't apply in that province. Always uncitable: the user has no way to verify without finding the source themselves. For an accessibility tool, a confidently wrong answer isn't neutral. It's actively dangerous.

Half a year ago, I wanted to see for myself what can we currently have with local LLMs. I went down the rabbit hole, learned quite a lot in the process, and shared my results in an article . The results were pretty discouraging: even with 32 GB VRAM, the best models I could run were both too slow and too dumb. At the same time, what you could get for free from inference providers was actually decent - and much faster. I remember my conclusion: "Let's wait for the next generation of models, which looks very promising. If we can run something comparable to full-size Qwen3-Coder-480B locally, that would be year of the Linux Desktop age of fully capable local LLMs. And now this day has arrived. Models Half a year later, I'm revisiting this question. And this time, the whole situation has turned upside-down. Almost none of the providers still have free tier, and anything that's still free is barely good enough even for the simplest tasks. And is rate-limited all over. And on the local side, the next Qwen lineup is out. So, that's what I'm going to be looking at. Once again, I have two RX6800's, 16 GB each, and 64 GB RAM. On one hand, this is more VRAM than any "normal person" can have with one GPU - unless you've got something specifically for AI, like an unified-memory Mac or a DGX Spark. On the other hand, RX6800 is "pre-AI" - anything newer will have much better performance thanks to tensor processors. Qwen3.6-27B : This is a dense model, so basically you can't run it at all on anything less than 32 GB VRAM. It's the slowest one, but also the best one if you can run it. Its accuracy is claimed to be on par with Claude 4.5 Opus, and better than Qwen3.5-397B-A17B . This is what I've been waiting for. It runs reasonably fast on my setup, so it's very much usable both in terms of performance and accuracy. Qwen3.6-35B-A3B : This one is MoE, and it's pretty small, so it's the fastest one. It's good for anything that doesn't require too much (i.e. for agentic tasks that don'

I've noticed something about myself. It's become much harder to get things done. I see less movement compared to my previous self. And sometimes, when I see people get so much done, tasks, projects, anything, I wonder why I can't make progress like that. Now I'm not comparing myself to Elon Musk here, I'm just wondering if I'm doing my best. For the longest time, I've wanted to build a portfolio website. But for some reason or another, I never actually ended up making it, or should I say, starting it. Whenever I had a free block of time, I wouldn't know where to start. I also have this side project I've been wanting to work on. I started it, then never made any more progress beyond the initial feasibility analysis. Why did I never allocate time to either? The Overthinking Starts Early I wanted the portfolio website to be really good. I wanted cool additions, beyond the usual "about me" and "projects" sections, I wanted things like "songs I'm obsessed with right now" and "last night I slept at." I was also looking if my fitbit has an API to show my live heart-rate on my portfolio (why would someone do that?). I was obsessed with the features, and with using the right tech to make it as efficient as possible. What stack should I even use for a portfolio? Same story with this very blog. Before writing a single word, I was wondering what stack to use. I looked into a ton of existing blogs for inspiration, how do I track views, what about comments, how do I stop a DDoS from flooding my DB if I'm storing comments, should I require auth just to leave a comment, should I support font and background color changes, themes, what's the best tool to do all of that? I've noticed this pattern with almost everything I do. I'm so obsessed with doing things right on the first attempt that I never actually make the first attempt, because "doing it right" is just too demanding. Or a related trap, when I want to do something perfectly , I discover prerequisites, and those prerequisites

You wire up the OpenAI SDK, ship the feature, and it works. Three weeks later someone in finance forwards a screenshot of a bill that tripled and asks what happened. You open the provider dashboard, see one big number, and… that's it. No per-feature breakdown, no idea which change caused it, no way to tell whether it's a bug or just growth. I've watched this happen at enough teams that I now treat "we can't explain our AI bill" as a predictable stage every company hits about two months after their first LLM feature ships. Here's how to get ahead of it — starting with plain code, then the tradeoffs, then where a dedicated tool actually earns its keep. Disclosure up front: I work on StackSpend, which does the full version of this. I've kept the first 80% of this post vendor-neutral because most of it you can and should build yourself before you buy anything. The core problem: the bill is a single number, your costs are not Provider dashboards give you total spend over time. What you actually need to make decisions is spend broken down by the dimensions you care about: Per feature — is it the summarizer or the chat assistant that's expensive? Per customer / tenant — which accounts cost more to serve than they pay? Per model — how much are you spending on GPT-4-class vs cheaper models? Per environment — is a runaway staging job quietly burning money? None of those dimensions exist in the raw bill. You have to attach them yourself, at call time, because after the request is gone the context is gone with it. Step 1: capture usage at the call site Every major provider returns token usage in the response. The trick is to log it with your own business context attached — the feature name, the tenant, the environment. Here's the pattern in TypeScript with the OpenAI SDK: import OpenAI from " openai " ; const openai = new OpenAI (); // Prices per 1M tokens — keep these in config, they change often. const PRICING : Record < string , { input : number ; output : number } > = { " g

When I published the first article about the Neuron AI Router , I expected questions about routing rules. Which rule to use for structured output, how to write a custom one, how the round robin behaves under load. Some of those questions arrived, but the most frequent one was different, and it wasn't really about routing at all. It was about failure. What happens to my agent when the provider goes down? It is a fair question, and if you are new to building AI applications it deserves a proper answer before we look at any code. Here is the short version. The new fallback strategy in Neuron AI Router lets you define an ordered list of LLM providers for your PHP agent. When an inference call fails with a transient error, such as a rate limit, a timeout, or an overloaded server, the same request is automatically retried on the next provider in the list. The failover is transparent: the agent never knows it happened, and the conversation continues without losing state. The rest of this article explains why this problem exists, why the usual solutions fall short, and how to configure it. Why LLM providers fail in production An LLM provider is an external service you talk to over HTTP. Every time your agent thinks, it is making a network call to a machine you don’t control, operated by a company that is currently serving millions of other requests. These services fail in very ordinary ways. You hit a rate limit because your traffic spiked. The provider returns an “overloaded” error because their traffic spiked. A request times out. A deployment on their side causes a few minutes of elevated error rates. None of this means you did something wrong, and none of it is rare. If you keep an agent in production long enough, you will see all of these. In a classic web application, a failed call to a third party API is usually a corner of the system. You log it, maybe retry it in a queue, and the rest of the page still works. In an agent based application the inference call is not

Spanlens is an open-source (MIT) LLM observability platform that lets developers monitor every call their application makes to OpenAI, Anthropic, Gemini, Mistral, OpenRouter, Azure OpenAI, or a local Ollama model. Integration takes one line: swap your client's baseURL to the Spanlens proxy, or run "npx @spanlens /cli init" and the wizard rewrites your code automatically. From that moment, every request is recorded with its model, token counts, latency, cost, and full prompt and response body, with streaming responses reconstructed automatically. The dashboard turns that raw log into operational insight. Cost tracking breaks spend down per request, per model, and per end user, and parses prompt-cache tokens separately so you see real cache savings rather than sticker price. Agent tracing visualizes multi-step workflows as Gantt waterfalls and node-and-edge graphs, highlighting the critical path so you can find the slowest dependency chain in a fan-out. Anomaly detection flags 3-sigma deviations in latency, cost, or error rate against a rolling 7-day baseline with root-cause hints. Alerts on budget, error rate, and p95 latency are delivered to Email, Slack, or Discord. Spanlens goes beyond passive logging. A regex-based PII and prompt-injection scanner inspects request and response bodies and can block injections at the proxy. The savings engine spots calls that match a cheaper model's profile (for example, a gpt-4o call that looks like a classification task) and estimates the monthly saving from switching. Prompt versioning with A/B experiments compares versions on latency, cost, and error rate using Welch's t-test for statistical significance, and an LLM-as-judge evaluation framework (judge with OpenAI, Anthropic, or Gemini) scores outputs against rubric anchors, with human agreement measured by Pearson r or Cohen's kappa. Reusable datasets power offline evals and regression checks.

At the beginning of June I started a “ Claude withdrawal ” challenge. The plan was to run MiniMax 3 for a month, to see if I can get the same level of quality, but at 5x less the price. Until then, Claude Code was my main driver, with MiniMax on the backup, for when I was running out of quota, or sometimes for code review. The monthly bill for Claude was $100 on the Max plan, whereas for MiniMax I would pay $20 for the Token plan. All in all, it seemed like an interesting experiment. Then, half way through the challenge, Grok came into the picture. I got a very interesting offer at $35 for 3 months, then $35/month. But Grok has something neither Claude, nor MiniMax can give me out of the shelf: video and image generations. The only unknown was if switching from Claude Code to Grok will still maintain the same coding power. So I instantly took the offer, and did whatever I had to do to understand if this was the right path. And here comes the “whatever I had to do”, in plain technical terms. Switching from Claude Code to Grok – the Actual Steps The switch itself was interesting because I didn’t want to lose the Claude Code interface. I like the harness. The way it works with my codebase, the commands, the flow. So I used a helper called cliproxyapi . It’s a small proxy that sits between the Claude Code client and whatever model you point it at. You run it locally, tell it to forward requests to Grok’s API instead of Anthropic’s. Then you launch Claude Code the same way you always do, but it talks to Grok under the hood. Here’s how it goes in practice. Step 1: Install the proxy. I used brew to install it, I’m on a Mac, and also because I wanted to have it started as a service. Step 2: Set two environment variables. One is the target API base URL, for Grok that’s something like https://api.x.ai . The other is your API key. "env" : { "ANTHROPIC_BASE_URL" : "http://localhost:8317" , "ANTHROPIC_API_KEY" : "cliproxy-local-key" } , Notice how we use “cliproxy-local-key”, be