标签：#llm

If your LLM costs are climbing, the instinct is almost always the same: swap to a cheaper model. GPT-4 to GPT-4-mini. Claude Opus to Claude Haiku. Sometimes that helps a little. It rarely fixes the actual problem. The actual problem, in most workflows I've looked at, is that every step gets routed through the LLM, even the steps that don't need language reasoning at all. This post breaks down a simple mental model for deciding what should and shouldn't touch an LLM, with a working example you can adapt. The four components of any AI workflow Every automated workflow — whether it's a support ticket router, a fraud check, or a content pipeline — is built from some combination of four building blocks. They get treated the same once a workflow diagram is drawn flat, but they have wildly different cost and latency profiles. Component What it does Think of it as Typical cost Trigger Starts the workflow The doorbell ~$0 Deterministic ML Structured predictions — classify, score, rank The calculator Cents per 1,000 calls LLM / Generative Reads, writes, reasons in language The writer Dollars per 1,000 calls Tool / API Fetches or writes real data The hands Cents per 1,000 calls The gap between row 2 and row 3 is the whole article. A classifier and an LLM call can solve the exact same problem, but one costs roughly 100-1000x more than the other, depending on model and provider. If you're not deliberately deciding which one handles which step, you're probably defaulting to the expensive one — because in frameworks like LangChain or a quick custom agent loop, it's just easier to shove everything into a prompt. Where this actually shows up Here's a workflow I see constantly: an automated support ticket triage system. flowchart LR A[New support ticket] --> B{Classify intent} B --> C[Route to team] B --> D[Auto-draft response] D --> E[Update CRM] A naive build sends the entire ticket text to an LLM and asks it to do everything at once: classify the intent, decide routing, draft a re

There's a quiet assumption baked into a lot of agent code: that a bigger context window means a better memory. Vendors ship 200K, then 1M, then 2M token windows, and the implied promise is "just put everything in and the model will remember." After building agents that run for weeks, I've come to think this conflates two things that are not the same — and treating them as the same is exactly why long-running agents get dumber over time. The context window is working memory. Real memory is what survives when the window is gone. Mixing them up is like confusing your desk with your filing cabinet. Two different clocks Working memory (the context window) lives for one session, maybe one turn. It's fast, expensive, and volatile. It's where reasoning happens right now . Durable memory lives across sessions. It's slow, cheap, and persistent. It's what the agent knows when it wakes up tomorrow with an empty window. These have different lifespans, different costs, and different access patterns. The moment you try to make one do the other's job, things break: Use the window as memory → everything you "remember" has to be re-loaded every turn, you pay for it every turn, and the instant the session ends it's gone. Use durable storage as working memory → you're reading and writing files mid-reasoning for things that only matter for the next 30 seconds. A good agent keeps them separate on purpose. Why "just use a bigger window" fails Say you have a 1M token window and you stuff the entire history in. Three problems show up, none of which a bigger number fixes: Cost scales with every turn, not every session. That 1M tokens isn't paid once — it's re-sent on each step of a multi-turn task. A 20-step task can mean 20× the bill, mostly re-reading the same stale history. Attention dilutes. "Lost in the middle" is real: models attend most reliably to the start and end of a long context. Bury the one fact that matters under 900K tokens of transcript and recall quality drops, even though

Code generation is solved, but memory isn't. Here's an argument for why the SDLC is inverting with intent becoming the spine and code becoming a layer you drill into, explaining what teams lose every time an agent's reasoning disappears.

I'm 16, from Pune, India. For the past couple of years I've been building O-AI — a fully local AI desktop assistant. No cloud. No API keys. No data leaving your machine. Everything runs on your own GPU. Why I built it Every AI assistant I tried sent data somewhere. ChatGPT, Copilot, Gemini — all cloud. I wanted something that felt like JARVIS from Iron Man: smart, fast, personal, and private. So I built it from scratch. What O-AI can do Core engine: Runs LLMs fully on-device via llama.cpp / Ollama (zero internet required) Self-learning core — extracts facts from every conversation and stores them permanently Fine-tuning pipeline — train the model on your own data, locally Voice & language: Voice control in English, Hindi, and Marathi via Whisper (running locally) Responds in whatever language you speak Modes: JARVIS mode — arc-reactor HUD, 4 reactive states, British-male voice, "sir" persona Take Over PC mode — full desktop automation Animated floating desktop pet (4 types, draggable, reacts to voice) 30+ automation fast-paths: open apps, search the web, control media, screen vision, run code, edit files, cursor control, social media steps, clipboard ops... Multi-step agent system: plan → execute → verify loop with 14+ step types (web_search, fetch_url, read_screen, run_code, edit_file, open_social, and more) Stack Backend: Python (Flask IPC + agent core) Frontend: Electron + vanilla JS LLM: llama.cpp / Ollama Voice: Whisper (local) + Edge TTS / neural voice Vision: PIL + screen capture The hardest bugs "Says done but isn't" — Early versions reported success even when an agent step failed. Fixed by building a proper outcome verifier that reads the actual result, not the plan. The "opens a random video" bug — Asking the agent to play something would open random YouTube videos. Root cause: the plan validator wasn't catching placeholder URLs like [video_url] . Fixed with a universal content guard on all plans. GPU offloading on Windows — Getting all 32 layers onto the

I spent a week building Wingman , an open source MCP server that renders a persistent task panel inline in Claude conversations using MCP Apps (SEP-1865). The spec is solid. The SDK is solid. But I hit two bugs that cost me most of a weekend each, and neither is documented anywhere I could find. Writing them up here in case they save someone else the time. Bug 1: resourceUri has two valid-looking locations, and only one works MCP Apps needs a way to tell the host "render this resource as a UI for this tool call." That pointer lives in _meta.ui.resourceUri . The question is: meta on what? I started with a parameterized resource template, ui://wingman/panel/{plan_name} , registered per plan. That was my first mistake. Parameterized templates get listed under resources/templates/list , not resources/list , and hosts do not prefetch or render anything from the templates list. The fix was straightforward once I found it: register one static resource, ui://wingman/panel , and pass the actual plan data through structuredContent on the tool result instead of baking it into the URI. That fix surfaced the real bug. My show_plan tool was returning a plain Python dict: return { " plan " : plan_data , " _meta " : { " ui " : { " resourceUri " : " ui://wingman/panel " }} } This looks correct. It is not. FastMCP's result conversion takes a returned dict and serializes the whole thing into structuredContent , verbatim, including any _meta key the dict happens to carry. So the actual wire result looked like this: result . structuredContent [ " _meta " ][ " ui " ][ " resourceUri " ] # == "ui://wingman/panel", but wrong place result . meta # None — this is what the host actually reads MCP Apps hosts read resourceUri off the top-level _meta on the CallToolResult , not off whatever ended up inside structuredContent . With that pointer effectively missing, the host had nowhere to bind the iframe. The visible symptom was strange: actions in the UI would update on screen but nothing persist

"It runs on my own GPU, so it's basically free." I believed that until I put a meter on it. So I ran a controlled benchmark on one box — an openSUSE machine with a single RTX 3090 — driving three local models through ollama under an identical fixed workload (256-token generations in a loop for ~4 minutes each), while my open-source dashboard priced every run by the real GPU energy it burned : power sampled from nvidia-smi every 10 s, integrated over each run's exact window, multiplied by my actual day/night tariff. One number per model, in euros per million output tokens. Here's the part that made me re-run it. The tiny gemma3:1b came out at €0.118 / 1M tokens — about 5× cheaper than a hosted Flash-class API (~€0.55). But gemma3:27b 's electricity alone was €0.706 / 1M — more expensive per token than just paying the cloud, and that's before a single cent of the GPU's purchase price. "Local" didn't make it cheaper; it made it cost more and I own the depreciation. The mechanism is one line: each token costs watts ÷ throughput , and a big dense model is both slow and thirsty. A newer mid-size architecture ( gemma4:26b ) bought a lot of that back, landing at €0.272 . The full guide is methodology-first and reproducible end to end — minting an ingest key, the stdlib-only client, the exact ollama loop that reads eval_count / eval_duration for real tokens-per-second, reading each run back priced, and the honest caveats (this is marginal GPU energy only — not capex, idle, or cooling — and the absolute numbers round to fractions of a cent; the shape is the finding). Read the full guide on Medium → https://medium.com/@arsen.apostolov/how-much-does-it-actually-cost-to-run-a-local-llm-per-million-tokens-measured-4a90a7f31a48

I wanted to get better at prompt engineering. Not the trick-the-robot kind, the boring-but-useful kind: how to ask a model a question so you get an answer you can actually trust. The trouble with practicing is that most tutorials use made-up examples, and it's hard to tell a good answer from a bad one when you don't care about the topic. So I practiced on something I did care about: the deals sitting in my Amazon cart. I had a vacuum I'd been eyeing and a hair styler that was "43% off," and I genuinely wanted to know if those were good prices or just good marketing. The stakes were real, actual money on an actual decision, and that's what made it a good drill. A vague prompt gives you a confident answer, and when you actually care, you can feel that the answer is hollow. What I learned, with the real deals and the actual before-and-after prompts: The trap hiding in every deal Start with the hair styler. The listing said: Shark FlexStyle. Limited time deal. $199.00, 43% savings. List Price: $349.99. My first instinct was the prompt most people write: "Shark FlexStyle $199, 43% off list $349.99, is that a good deal?" This feels reasonable. It is also nearly useless: it lets the model answer the easy question (is 43% off a big discount? sure!) instead of the real one (is $199 actually a good price?). That $349.99 list price is a marketing anchor. A lazy prompt accepts it, and so you get a lazy "yes, great deal!" back. The fix was re-framing this: Act as a pricing analyst. I don't care whether $199 looks like a discount off list. I care whether $199 is a genuinely good price for the Shark FlexStyle right now. Before concluding, work through: (1) the actual street price over the last 6-12 months, (2) how often it drops to or below $199, (3) the real discount vs. its typical selling price, not vs. list. Cite a source and date for each price, or mark it unverified. Same question, completely different answer. What the assistant came back with, in its own telling: $199 is a

Nvidia's Nader Khalil — Director of Developer Technologies and co-founder of Brev.dev, acquired by Nvidia two years ago — sat down with The New Stack to talk agents, OpenClaw, and where enterprise AI is heading. His opening line is worth keeping: "An agent is an LLM and a harness. And if you think about that, it involves two things. It involves the loop and the LLM… Each loop should take us closer to our goal." That's not a complicated definition. It's also exactly right — and the fact that Nvidia's internal framing lands here matters more than the quote itself. What actually happened Nvidia has full-time OpenClaw contributors. Khalil: "We have a couple of developers at the company that contribute to OpenClaw full time." That's a real commitment, not a press-release mention. NemoClaw is their enterprise blueprint — a reference architecture for running OpenClaw (and Hermes) in production, with GPU routing, security policies, and a runtime called OpenShell. Khalil traces the harness evolution directly: from ChatGPT's system prompts → memory → file context → Cursor → Claude Code. All of it is harness, not model. The model is constant; the harness is where the product lives. On OpenClaw's PR backlog: "It got more stars than Linux in months… so I think you're gonna see a mountain of PRs." Their response — roll up their sleeves and start merging. Why this framing matters Nvidia makes money when AI compute scales. For that to happen, agents need to work reliably in enterprise environments — and the harness is the reliability layer. Their NemoClaw blueprints aren't a product play; they're an enablement play. Enterprise teams get a reference architecture that works on Nvidia silicon. Nvidia gets demand for the GPUs underneath. It's the CUDA X model applied to agentic AI. The microwave analogy Khalil uses is useful: "when it's your microwave at home, you just go 'Boop, boop. Done.'" Every enterprise will build specialized agents tuned to their domain — CrowdStrike, Cadence, P

No fancy infrastructure. Just a 50-line Python function that picks the right model for the right query. Last month my LLM API bill hit $340. This month: $67. Same traffic. Same product. The only change was adding a simple router that stops sending every request to Claude Sonnet when GPT-4o mini can handle it just as well. Here's exactly how it works. The Problem When you prototype, you pick one model and hardcode it everywhere. Usually something capable like GPT-4o or Claude Sonnet, because you want good results fast. Then you ship, traffic grows, and you get a bill that makes you question your life choices. The thing is — not all queries need a flagship model. In a typical RAG app: "What is the return policy?" → GPT-4o mini handles this fine "Summarize these 5 conflicting documents and identify the key disagreement" → needs Sonnet You're paying Sonnet prices for return policy questions. That's the bug. The Fix: A Complexity Router import anthropic from openai import OpenAI openai_client = OpenAI() anthropic_client = anthropic.Anthropic() def classify_complexity(query: str) -> str: """Returns 'simple' or 'complex'.""" simple_indicators = [ len(query.split()) < 15, query.endswith("?") and query.count("?") == 1, not any(w in query.lower() for w in [ "compare", "analyze", "summarize", "explain why", "difference between", "pros and cons", "evaluate" ]) ] return "simple" if sum(simple_indicators) >= 2 else "complex" def route(query: str, context: str = "") -> str: complexity = classify_complexity(query) if complexity == "simple": # $0.15/M input — GPT-4o mini response = openai_client.chat.completions.create( model="gpt-4o-mini", messages=[ {"role": "system", "content": context}, {"role": "user", "content": query} ] ) return response.choices[0].message.content else: # $3.00/M input — Claude Sonnet (only when needed) response = anthropic_client.messages.create( model="claude-sonnet-4-6", max_tokens=1024, system=context, messages=[{"role": "user", "content": query}] ) retur

Cross-posted from the LLMKube blog . A local 27B coding model, running on hardware in my house, is a coin flip. Some runs it nails the fix in twenty minutes. Some runs it edits the wrong file, writes a test that passes no matter what the code does, and tells you it's done. The bet behind LLMKube's Foreman was never that I would find a local model good enough to trust. It was that I could build a harness I trust more than any single model's output. This weekend tested that bet harder than any benchmark could, because the harness spent the weekend building its own guardrails. Here is the short version of what happened across 0.8.12 and 0.8.13. My local coder built three new gates for itself. One of them shipped with the exact flaw it was written to catch, and the review caught it. Three new contributors sent four clean pull requests while the machines worked. The same model ran on an AMD box and an Apple Silicon Mac, and the Mac quietly won a round nobody expected. And not one byte of any of it touched a cloud API. The thesis, stated plainly Trust the harness, not the model. A coding agent on a local model produces output of wildly variable quality, and no amount of prompt tuning makes a 27B as reliable as a frontier model. So Foreman does not ask the model to be reliable. It wraps the model in a pipeline that is : the coder works in a cloned workspace, a fast in-workspace gate runs gofmt, vet, build, lint, and the unit tests for the packages it touched; a reviewer reads the diff against the issue; and a clean-room Kubernetes Job re-runs the full suite before anything is allowed to call itself a GO. Around all of that sit deterministic rails: scope checks, edit-free-streak detection, repo-map context. The model is a stochastic component inside a system whose job is to make the system's verdict trustworthy even when the component is not. The interesting question is never "is the model good." It is "does the harness catch the model when it is bad." This weekend gave me

The company warned about dangers of advanced AI far more than rival OpenAI.

A few days ago I had an idea: what if one LLM could orchestrate other LLMs as agents — not just calling them, but verifying that each agent's output was actually correct before passing it to the next? I work on NeuralBridge (an open-source self-healing SDK for LLM pipelines), so I decided to build it and test it with two real providers: KIMI (Moonshot) and Agnes AI . The Core Problem: Failover ≠ Correctover Most API gateways and LLM routers stop at "HTTP 200" — they retry or switch providers, but they never check if the output is actually correct . # What everyone else does: try : result = call_llm ( prompt ) return result # HTTP 200 = success? 🚩 except Exception : result = call_llm_fallback ( prompt ) return result # Still not verified! This is dangerous. A failover from gpt-4o to gpt-4o-mini might silently drop 3 critical fields. A KIMI response that returns "200 OK" might still be missing key entities. Correctover is the idea that switching providers isn't enough — you must verify semantic equivalence after every switch. The Architecture We built a simple DAG-based chain executor with three key capabilities: DAG orchestration — define multi-step workflows where nodes depend on each other Per-node semantic validation — every LLM output is checked against a Contract before passing to the next node Cross-provider Correctover — if validation fails, automatically retry with a different provider from neuralbridge import SelfHealingEngine , ProviderConfig , Contract from neuralbridge.chain import ChainBuilder engine = SelfHealingEngine ( providers = []) engine . add_provider ( ProviderConfig ( name = " moonshot " , base_url = " https://api.moonshot.cn/v1 " , api_key = " ... " , models = [ " moonshot-v1-8k " , " moonshot-v1-32k " ], )) engine . add_provider ( ProviderConfig ( name = " agnes " , base_url = " https://apihub.agnes-ai.com/v1 " , api_key = " ... " , models = [ " agnes-2.0-flash " ], )) chain = ( ChainBuilder ( engine ) . node ( name = " planner " , system = "

MCP is useful, but most of the time you do not actually need it. It gives an agent a clean way to discover tools, call APIs, and work with external systems. In practice, a skill file can describe the same usage path without dragging the whole MCP surface into context. But MCP is not free; rather than MCP itself, the real issue is the habit of loading a big MCP surface into every session, no matter what the session is actually about. Once a Claude Code or Codex run pulls in a bunch of servers, the model sees those tool definitions right away, even if the job is just writing docs or fixing a small bug. That is where the waste starts. The hidden cost of always-on MCP Every MCP server brings metadata with it: tool names, descriptions, argument schemas, nested parameters, enums, examples, and sometimes prompts or resources. While useful, this is still context. If you connect a handful of lightweight tools, the overhead is annoying but manageable. If you connect a real stack of services, the cost compounds fast. In practice, you end up paying for: tool discovery before the task starts schema text the model may never use repeated loading across unrelated sessions extra context pressure that pushes out the actual work That last point matters more than people think. Context acts as the active working set the model uses to reason. The more of it you burn on static tool catalogs, the less room you have for the user request, the repo state, prior reasoning, and the actual answer. Anthropic has already written about this problem directly in the context of MCP. Their engineering post on code execution with MCP calls out tool-definition bloat and shows how direct tool calls can consume a lot of context before the model even starts doing the real job. The tool list is not just setup noise; it is part of the session cost. Why skills are cheaper Skills take a different path. A skill file keeps the always-loaded portion tiny. Usually that means just the skill name and a short descript

A few months ago I was demoing my RAG-powered support bot to a colleague, feeling pretty confident about it. Then it confidently told her our refund policy was “30 days, no questions asked.” Our actual policy is 14 days, with conditions. The bot didn’t hedge. It didn’t say “I’m not sure.” It just made it up and said it with the same calm tone it uses for everything else. That demo stung. RAG was supposed to fix hallucinations, not just relocate them. Here’s what I learned debugging it, roughly in the order I learned it. 1. My chunks were too big, and too dumb I was splitting documents by character count, 1000 chars with slight overlap. It felt efficient. It wasn’t. A single chunk often contained unrelated sections. For example, the end of a “Shipping Policy” and the start of a “Returns Policy” could sit together in the same block. So when the retriever saw a query about returns, it would grab that chunk and the model would blend both sections into one confident but wrong answer. Fix: I switched to semantic chunking based on headings and paragraphs instead of raw character limits. More work upfront, but it stopped feeding the model Frankenstein context. 2. I trusted top-k similarity way too much My retriever was pulling the top 3 chunks by cosine similarity and passing them straight into the prompt. The problem: “similar” is not the same as “relevant.” A chunk can be semantically close to the query but still not actually contain the answer. The model doesn’t know that, it just assumes everything in context is true. Fix: I added a reranking step using a cross-encoder and started logging retrieval scores properly. That alone made it obvious when the system had no real answer but was still trying to act confident. 3. I never told the model it was allowed to say “I don’t know” My prompt was basically: “Use the context to answer the question.” That’s it. No instruction on what to do when the context is insufficient. So the model did what LLMs do when under-specified: it f

你知道吗？最近一个 AI Agent 直接删除了生产数据库，然后在 Twitter 上轻松"自首"——这条消息在 Hacker News 上获得了 860 分和超过 1000 条评论。随着 AI Agent 从演示走向生产环境，"在我的机器上能跑"和"它能安全地运行我的业务"之间的鸿沟从未如此巨大。 Pydantic AI 正是为弥合这一鸿沟而来。这个拥有 17,895 Stars 的 Python Agent 框架，由 Pydantic Validation 的同一团队打造——而 Pydantic Validation 正是 OpenAI SDK、Anthropic SDK、Google ADK、LangChain、LlamaIndex、CrewAI 等数十个 GenAI 工具的数据验证层。如果你用过 FastAPI，你已经知道 Pydantic 的感觉：类型安全、优雅、生产就绪。Pydantic AI 将这种哲学完整地带入了 Agent 开发领域。 在 2026 年的今天，大多数 Agent 框架把 LLM 当成返回字符串的黑盒子。Pydantic AI 则将其视为类型化、可验证、可组合的系统——这改变了一切。 隐藏用法 #1：人工审批工具调用（阻止 Agent 乱来） 大多数人的做法： 给 Agent 完全自由去调用任何工具——数据库查询、API 调用、文件删除——然后祈祷一切顺利。当出了问题时，你从用户口中才知道。 隐藏技巧： Pydantic AI 的延迟工具（deferred tools）让你可以标记某些工具调用在执行前需要人工审批。Agent 会规划好动作，但在执行前暂停并等待人类批准或拒绝——完美适用于数据库写入、金融交易或生产部署等破坏性操作。 from pydantic_ai import Agent , RunContext , DeferredToolRequests from pydantic import BaseModel agent = Agent ( ' openai:gpt-4.1 ' , output_type = str , # 通过 deferred=True 标记需要审批的工具 ) @agent.tool ( deferred = True ) async def delete_user_account ( ctx : RunContext , user_id : str ) -> str : """ 永久删除用户账户。需要人工审批。 """ # 这段代码只在人工审批通过后才会执行 await db . users . delete ( user_id ) return f " 用户 { user_id } 已删除 " # 运行 Agent——如果它尝试调用 delete_user_account， # 执行会暂停并返回 DeferredToolRequests 对象 result = await agent . run ( " 清理不活跃账户 " ) if isinstance ( result . output , DeferredToolRequests ): # 将待执行的工具调用展示给人类审批 for tool_call in result . output . calls : print ( f " Agent 想要执行: { tool_call . tool_name } ( { tool_call . args } ) " ) approved = input ( " 是否批准？(y/n): " ) if approved . lower () == ' y ' : # 带审批结果恢复执行 result = await agent . run ( " 已批准 " , message_history = result . all_messages (), deferred_tool_requests = result . output , ) 效果： Agent 可以自主处理安全操作（读取数据、生成报告），而在危险操作上自动暂停等待人工判断。不再有删库跑路的意外。 数据来源： Pydantic AI GitHub 17,895 Stars（通过 GitHub API 验证，最后推送 2026-06-21）。HN Algolia 搜索"pydantic-ai agent framework"获得 12 分讨论帖。"AI agent deleted production database"故事在 HN 上获得 860 分/1032 条评论（数据来源：HN Algolia API）。 隐藏用法 #2：基于图的多 Agent 工作流 + 类型安全状态 大多数人的做法： 构建线性

업무 자동화 시스템을 만들 때 가장 먼저 드는 질문이 있다. "에이전트 하나면 안 되나?" 단일 에이전트로 시작하면 구조가 단순하고 디버깅도 쉽다. 그런데 실제 업무 맥락에서는 단일 에이전트가 빠르게 벽에 부딪힌다. 이 글은 왜 여러 에이전트가 협업하는 구조가 필요한지, 그리고 그 구조를 실제로 어떻게 설계하는지를 기술적으로 짚는다. 단일 에이전트로 충분하지 않은 이유 단일 에이전트가 실패하는 지점은 복잡한 기능 탓이 아니라 컨텍스트 길이와 직렬 실행의 구조적 한계 때문이다. LLM 기반 에이전트에 하나의 긴 작업을 맡기면 세 가지 문제가 동시에 발생한다. 첫째, 컨텍스트 창이 소진된다. 데이터 수집, 변환, 검증, 발행을 하나의 루프에서 처리하면 중간 상태가 프롬프트에 누적되고, 모델은 앞부분 지시를 잊는다. 둘째, 직렬 실행은 병목을 만든다. API 호출이 5개 있고 각각 2초라면, 단일 에이전트는 10초를 기다린다. 셋째, 에러 격리가 불가능하다. 한 단계가 실패하면 전체 루프를 재시작해야 한다. 반면 여러 에이전트가 협업하는 구조에서는 각 에이전트가 명확한 책임 경계를 갖는다. 한 에이전트가 데이터를 수집하고, 다른 에이전트가 변환하고, 또 다른 에이전트가 검증한다. 에러는 해당 에이전트 범위 안에서 처리되고, 독립된 작업은 병렬로 돌린다. 에이전트 협업 구조를 어떻게 설계할까? 나무숲이 실제 자동화 프로젝트에서 가져가는 구조는 오케스트레이터-워커(Orchestrator-Worker) 패턴이다. 이 패턴은 Anthropic이 공개한 에이전트 설계 가이드라인 에서도 다루는 구조로, 책임 분리가 명확하다는 점이 핵심이다. 역할 책임 범위 주요 판단 오케스트레이터 전체 작업 계획, 워커 배정 어떤 워커를 호출할지, 순서와 병렬 여부 워커 에이전트 단일 도메인 작업 실행 도구 호출, 결과 반환 검증 에이전트 출력 품질 검사 재시도 요청 또는 다음 단계 진행 상태 관리 에이전트 간 공유 컨텍스트 보존 어떤 정보가 다음 에이전트에 전달되는지 예를 들어 콘텐츠 자동 발행 파이프라인이라면, 오케스트레이터가 "오늘 발행할 항목 목록"을 받아 수집 워커, 요약 워커, 포맷 워커를 순서대로 호출한다. 검증 에이전트는 포맷 워커 출력을 보고 발행 가능 여부를 판단한다. 에이전트 간 데이터 흐름과 오케스트레이션 설계 오케스트레이터는 각 워커를 직접 호출하고, 그 결과를 다음 워커의 입력으로 넘긴다. 이때 중요한 설계 결정이 두 가지다. 메시지 구조를 명시적으로 정의한다. 에이전트 간 데이터는 자유형 텍스트가 아니라 스키마가 있는 구조로 전달해야 한다. JSON 스키마나 Pydantic 모델을 쓰면 에이전트 출력이 다음 에이전트의 입력 형식을 충족하는지 런타임 전에 검사할 수 있다. 오케스트레이터는 작업의 의미를 이해하지 않는다. 좋은 오케스트레이터는 라우터에 가깝다. "이 입력은 A 워커에게, 그 결과는 B 워커에게"를 결정할 뿐, 각 작업의 도메인 로직에 관여하지 않는다. 이 원칙을 지키면 워커를 교체하거나 추가할 때 오케스트레이터 코드를 손댈 필요가 없다. 간단한 파이썬 예시로 구조를 보면 이렇다: from anthropic import Anthropic from pydantic import BaseModel client = Anthropic () class WorkerOutput ( BaseModel ): status : str # "success" | "retry" | "failed" payload : dict error_message : str | None = None def call_worker ( system_prompt : str , user_input : str ) -> WorkerOutput : response = client . messages . create ( model = " claude-opus-4-5 " , max_tokens = 1024 , system = system_prompt , messages = [{ " role " : " user " , " content " : user_input }],

Corro varios agentes respaldados por Bedrock en producción: análisis de documentos, emparejamiento de contenido, búsqueda en registros, búsqueda semántica. Esta es una pasada honesta sobre el OWASP Top 10 para aplicaciones LLM desde el lado de la implementación : el código de verdad que defiende cada riesgo y, igual de importante, las categorías donde mi respuesta es "parcial" o "todavía no". Primero el modelo de amenazas Un flujo de agentes es un pipeline: entrada no confiable → prompt → modelo → parseo → actuar. Cada flecha es una superficie de ataque. Antes de cualquier control, la pregunta más útil que me hice fue esta: ¿qué puede HACER de verdad un agente si el modelo está completamente manipulado? Mi respuesta dio forma a todo lo de abajo. Los agentes son mayormente-de-lectura : llaman a un modelo, leen filas acotadas de la base de datos, y escriben resultados de análisis con llave del usuario que pide. Sin shell, sin SQL arbitrario, sin llamada a herramientas. El radio de impacto es chico por construcción, que es el control más barato que existe. TL;DR, estatus honesto OWASP LLM Mi estatus El control LLM10 Consumo sin límite Fuerte Límite de tasa + cortacircuitos de costo mensual + topes de tokens por modelo LLM06 Agencia excesiva Fuerte (por diseño) Sin llamada a herramientas; mayormente-de-lectura; escrituras acotadas a quien llama LLM01 Inyección de prompt Parcial Contenido del usuario enmarcado como DATOS (delimitadores + preámbulo anti-inyección) LLM02 Divulgación de info sensible Parcial Limpieza de PII por regex antes del modelo; exclusiones auditadas LLM05 Manejo inadecuado de salida Parcial Validación de esquema + chequeos de fundamentación + sanear-antes-de-renderizar LLM07 Fuga del system prompt Parcial Registro versionado de prompts + regla anti-eco LLM08 Vector/Embedding N/A (todavía no construido) (nada) AuthN/Z + interruptor de apagado Fuerte Llave interna, gateo por cuota/gama, deshabilitado por agente LLM10 Consumo sin límite: empieza aquí, e

Over three posts I built three fine-tuned models for the same banking-intent task — full fine-tuning a 270M model , LoRA on 1.5B , QLoRA on 7B . They all landed around the same accuracy. Which raises an honest, slightly uncomfortable question: if a 270M model on my laptop already worked, why reach for a 7B model at all? The answer most "bigger is better" content skips For this task — you wouldn't. A good engineer picks the smallest model that clears the bar , not the biggest one available. The small model is cheaper to serve, runs in milliseconds, and you fully own it. Choosing the 7B here would be over-engineering. Reaching for a bigger model isn't a flex. It's a response to a requirement the small one can't meet. Here are the four cases where small stops being enough: 1. The task is genuinely hard Banking77 is easy — 77 fixed labels, short clean queries. Small models saturate it. But ask for reasoning ("which of these three issues is the primary one?"), open-ended generation (write the reply, don't just classify), or real nuance, and there's a capability floor that more parameters buy. No amount of fine-tuning gives a 270M model abilities it doesn't have. 2. You have little data I had ~10,000 labeled examples — plenty for a small model. With 50, a small model can't learn the task, but a 7B model already "knows" banking concepts from pretraining and only needs a nudge. Bigger models need less task data because they bring more prior knowledge. 3. You need one model for many tasks This is the quiet superpower of LoRA/QLoRA. A single frozen 7B base can host dozens of swappable adapters — intent classifier, reply writer, summarizer, sentiment — all from one ~5GB footprint in memory. The 270M is single-purpose. This is why companies serve hundreds of fine-tunes from one base model. 4. Accuracy compounds at scale 93% means 7 in 100 queries misrouted. At 10M queries/month, that's 700,000 mistakes. If each costs a support escalation, the 2–3 points a bigger model buys can

In Part 2 , LoRA let me fine-tune a 1.5B model by freezing it and training tiny adapters. But the frozen base still sat in memory in 16-bit (~3GB). Now I wanted to go to Qwen2.5-7B — and hit a wall that LoRA alone doesn't solve. The problem A 7B model is ~15GB in 16-bit precision. A free-tier T4 GPU has 16GB. It would barely load, with no room left to actually train. The QLoRA insight QLoRA asks the question that naturally follows from LoRA: the base is frozen and only ever read — so why store it in full precision? So you quantize the frozen base to 4-bit (NF4, a format tuned for how neural-net weights are distributed) and run the LoRA adapters on top in normal precision. The base shrinks dramatically; the trainable part stays small and precise. from transformers import BitsAndBytesConfig bnb_config = BitsAndBytesConfig ( load_in_4bit = True , bnb_4bit_quant_type = " nf4 " , # NormalFloat4 bnb_4bit_use_double_quant = True , # quantize the quant constants too bnb_4bit_compute_dtype = torch . float16 , # dequantize to fp16 for the matmuls ) model = AutoModelForCausalLM . from_pretrained ( MODEL_ID , quantization_config = bnb_config , device_map = " auto " ) Each flag earns its place: load_in_4bit — store frozen weights in 4 bits instead of 16. nf4 — a 4-bit type matched to the bell-curve distribution of neural-net weights (better than plain int4). double_quant — quantize the quantization constants too, for a bit more savings. compute_dtype — dequantize to fp16 for the actual matmuls, so storage is 4-bit but compute stays precise. The moment it clicked One line of output: loaded in 4-bit. footprint: 5.44 GB I downloaded 15.2GB of weights and they sat in memory as 5.44GB. A model that couldn't be loaded for full fine-tuning was now training on a single consumer GPU — with room to spare. (The download is still 15GB; bitsandbytes quantizes on the fly during load.) The QLoRA-standard recipe Two more pieces beyond Part 2's LoRA setup: prepare the quantized model for trainin

I run a homelab with four RTX 3090s — 96 GB of VRAM, 44 CPU cores. For two weeks I tried to make it my daily driver for local LLM inference instead of paying for cloud APIs. I got it working. Then I looked at the numbers and subscribed to a paid API anyway. Here's the uncomfortable part, and the optimizations that still made it worth doing. ## The setup 4× RTX 3090 (Ampere — no native BF16), 96 GB VRAM total, 44 cores Models: Qwen3.6-35B-A3B (Q8_0, MoE) and Qwen3-Coder-Next (Q6_K, hybrid) llama.cpp in router mode + OpenWebUI Ceiling I hit: ~105 tokens/second ## The 6% problem The wall wasn't compute. GPU utilization sat at 6%. The bottleneck was CPU orchestration — llama.cpp dispatches across multiple GPUs sequentially, so the cards spent 94% of the time idle waiting on each other. Throwing more VRAM at it does nothing for this. ## What actually moved the needle | Change | Effect | |---|---| | --ubatch-size 512 | +40% throughput | | KV cache quantization (Q4_0) | 4× VRAM savings | | Speculative decoding (n-gram) | 2.5× speedup on repetitive tasks | | YaRN rope scaling | context extended to 1M tokens | Two things surprised me: MoE models tolerate aggressive quantization far better than dense ones — inactive experts don't eat bandwidth, so the quant hit lands softer. The 3B active -parameter model was great at local decisions but fell apart on coherence past ~300–400 lines of code — fine for a function, not for cross-file consistency. ## The conclusion I didn't want At ~11 kWh/day, plus hardware depreciation, against current API pricing, the math doesn't favor local for interactive work. The single biggest improvement to my daily AI workflow was paying for an API. Local still wins for privacy, high-volume batch jobs, or uncensored experimentation — but not as a general cloud replacement. It's an economics problem, not a capability one. I wrote up the full cost breakdown and the exact llama.cpp router configs on aipster.com . If you're weighing a local rig, I also benc